Skip to content

Commit ea3e182

Browse files
radicalradical
authored
[tests] Add support for running EndToEnd tests on helix with the aspire workload (#2534)
# Problem: 1. We want to run some EndToEnd tests on CI, which can `dotnet run` an aspire project, and allow individual tests to interact with the services. This requires: - Ability to build, and run an aspire project - IOW, a sdk with the `aspire` workload installed. - `docker` 2. Also, allow using `TestProject.*` in `tests/testproject`, in two modes: - `in-tree` test run which directly reference aspire projects, and repo targets - `out-of-tree` test runs which uses a SDK+workload based on local build output # Solution: ### SDK+workload - SDK with version from `global.json` is installed into `artifacts/bin/dotnet-{none,latest}` - The `aspire` workload is installed into the sdk using the nuget packages in `artifacts/packages` with a version like `8.0.0-{dev,ci}` - this also allows running tests that do `dotnet new aspire-starter; dotnet run` same as a user - this utilizes an internal nuget from `dotnet/runtime` ### TestProject - This can switch between the two test run modes using the msbuild property `$(TestsRunningOutsideRepo)` - when running `in-tree` the test project directly references hosting targets, and aspire projects via `ProjectReference` - when running `out-of-tree` the `ProjectReferences` and imports are replaced with `PackageReferences` to the Aspire nugets - Default is to run `in-tree` ### Helix - The tests are built on the build machine - The testproject, and the sdk+workload is sent to helix - where the tests run using `dotnet` from the sdk+workload path - Since `docker` is needed to helix, this is enabled only for `Linux` in this PR. Blocked on dotnet/dnceng#2067 for windows support. ## TL;dr or How do I use this? ### Steps to prepare for `out-of-tree` runs 1. `.\build.cmd -pack` - to build all the nugets 2. `dotnet build tests/Aspire.EndToEnd.Tests/Aspire.EndToEnd.csproj /t:InstallWorkloadUsingArtifacts /p:Configuration=<config>` - this will install the sdk, and the `aspire` workload using the nugets from `artifacts/packages` - at this point the sdk is usable outside the repo too. Note: you need to add `artifacts/packages/$(Configuration)/Shipping` as a nuget source so the locally built packages can be picked up. ### Using it from VS - No difference when running the EndToEnd tests in `in-tree` mode - For switching to `out-of-tree` add `<TestsRunningOutsideRepo>true</TestsRunningOutsideRepo>` to `tests/Aspire.EndToEnd.Tests/Directory.props` *before* any imports. - tests cannot be run at this point as they will fail complaining about `artifacts/bin/dotnet-latest` being missing - Install the sdk+workload following the steps above - Run/debug the tests normally now, and they will be using the sdk - Also note that in this case the testproject is run from the bindir for `Aspire.EndToEnd.Tests`, so a path like `artifacts/bin/Aspire.EndToEnd.Tests/Debug/net8.0/testassets/testproject/` ### Using it from command line - No difference when running the EndToEnd tests in `in-tree` mode - Install the sdk+workload following the steps above - When running the tests you can either: - set `<TestsRunningOutsideRepo>true</TestsRunningOutsideRepo>` to `tests/Aspire.EndToEnd.Tests/Directory.props` before any imports - or set the environment variable `TestsRunningOutsideRepo=true` ### Other details - `tests/Shared/WorkloadTesting` - has sources essentially from `dotnet/runtime`'s `Wasm.Build.Tests` - TestProject supports skipping individual resources like `--skip-resources oracledatabase,cosmos` - `.runsettings` with 15min timeout - `xunit.runner.json` - to get diagnostic messages - The core infrastructure is not specific for `Aspire.EndToEnd.Tests`, and will get used by other test projects in the future ###### Microsoft Reviewers: [Open in CodeFlow](https://microsoft.github.io/open-pr/?codeflow=https://github.com/dotnet/aspire/pull/2534)
1 parent a6581e5 commit ea3e182

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+1552
-207
lines changed

‎Directory.Build.props‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,12 @@
2424
<Nullable>enable</Nullable>
2525
<ImplicitUsings>enable</ImplicitUsings>
2626
<!-- Set RunSettingsFilePath property which is read by VSTest. -->
27-
<RunSettingsFilePath>$(RepositoryEngineeringDir).runsettings</RunSettingsFilePath>
27+
<RunSettingsFilePath Condition="'$(RunSettingsFilePath)' == ''">$(RepositoryEngineeringDir).runsettings</RunSettingsFilePath>
2828
<!-- We don't want to use the workload for AppHost projects in this repo -->
2929
<SkipAspireWorkloadManifest>true</SkipAspireWorkloadManifest>
3030

3131
<DashboardPublishedArtifactsOutputDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsDir)', 'DashboardArtifacts', '$(Configuration)'))</DashboardPublishedArtifactsOutputDir>
32+
<WorkloadsPackageSource>$(ArtifactsShippingPackagesDir)</WorkloadsPackageSource>
3233
</PropertyGroup>
3334

3435
<!-- OS/Architecture properties for local development resources -->
@@ -42,6 +43,7 @@
4243
<DcpDir>$(NuGetPackageRoot)microsoft.developercontrolplane.$(BuildOs)-$(BuildArch)/$(MicrosoftDeveloperControlPlanedarwinamd64PackageVersion)/tools/</DcpDir>
4344

4445
<TestArchiveTestsDir>$([MSBuild]::NormalizeDirectory($(ArtifactsDir), 'helix', 'tests'))</TestArchiveTestsDir>
46+
<TestArchiveTestsDirForWorkloadTests Condition="'$(TestArchiveTestsDirForWorkloadTests)' == ''">$([MSBuild]::NormalizeDirectory($(ArtifactsDir), 'helix', 'workload-tests'))</TestArchiveTestsDirForWorkloadTests>
4547
<!-- TODO: Need to figure out we can automatically detect target framework here. This property
4648
is specified to support dashboard path metadata generation on the inner loop. -->
4749
<AspireDashboardDir>$(MSBuildThisFileDirectory)/artifacts/bin/Aspire.Dashboard/$(Configuration)/net8.0/</AspireDashboardDir>

‎Directory.Build.targets‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
<ReadMePath>$(MSBuildProjectDirectory)\README.md</ReadMePath>
55
<ReadMeExists Condition="Exists('$(ReadMePath)')">true</ReadMeExists>
66
<PackageReadmeFile Condition="'$(PackageReadmeFile)' == '' And '$(ReadMeExists)' == 'true'">README.md</PackageReadmeFile>
7+
8+
<TestArchiveTestsDir Condition="'$(IsWorkloadTestProject)' == 'true'">$(TestArchiveTestsDirForWorkloadTests)</TestArchiveTestsDir>
79
</PropertyGroup>
810

911
<PropertyGroup>

‎Directory.Packages.props‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
<Project>
22
<!-- Workaround https://github.com/dependabot/dependabot-core/issues/8490 -->
3-
<Import Project="eng/Versions.props" Condition="'$(MajorVersion)' == ''"/>
3+
<!-- This file gets imported for out-of-tree test runs also where eng/Versions.props isn't
4+
available -->
5+
<Import Project="eng/Versions.props" Condition="'$(MajorVersion)' == '' and Exists('eng/Versions.props')"/>
46
<PropertyGroup>
57
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
68
</PropertyGroup>
@@ -126,6 +128,7 @@
126128
<!-- unit test dependencies -->
127129
<PackageVersion Include="bUnit" Version="1.26.64" />
128130
<PackageVersion Include="Microsoft.Extensions.Diagnostics.Testing" Version="$(MicrosoftExtensionsDiagnosticsTestingPackageVersion)" />
131+
<PackageVersion Include="Microsoft.NET.Runtime.WorkloadTesting.Internal" Version="$(MicrosoftNETRuntimeWorkloadTestingInternalVersion)" />
129132
<PackageVersion Include="Testcontainers.RabbitMq" Version="3.7.0" />
130133
<!-- Pinned version for Component Governance - Remove when root dependencies are updated -->
131134
<PackageVersion Include="Newtonsoft.Json" Version="13.0.3" />

‎NuGet.config‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,23 @@
1212
<!-- Begin: Package sources from dotnet-runtime -->
1313
<!-- End: Package sources from dotnet-runtime -->
1414
<!--End: Package sources managed by Dependency Flow automation. Do not edit the sources above.-->
15+
<!-- TEST_RESTORE_SOURCES_INSERTION_LINE -->
1516
<add key="dotnet-public" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json" />
1617
<add key="dotnet-eng" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json" />
1718
<add key="dotnet8" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet8/nuget/v3/index.json" />
1819
<add key="dotnet-libraries" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-libraries/nuget/v3/index.json" />
1920
<add key="orleans-nightly" value="https://orleans.pkgs.visualstudio.com/orleans-public/_packaging/orleans-nightly/nuget/v3/index.json" />
21+
<add key="dotnet9-transport" value="https://dnceng.pkgs.visualstudio.com/public/_packaging/dotnet9-transport/nuget/v3/index.json" />
2022
<add key="azure-sdk-for-net-dev" value="https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json" />
2123
</packageSources>
2224
<packageSourceMapping>
25+
<!-- used for installing test workloads for resolving aspire workload manifest -->
26+
<packageSource key="nuget-local">
27+
<package pattern="*Aspire*" />
28+
</packageSource>
29+
<packageSource key="dotnet9-transport">
30+
<package pattern="*WorkloadBuildTasks*" />
31+
</packageSource>
2332
<packageSource key="azure-sdk-for-net-dev">
2433
<package pattern="Azure.Provisioning" />
2534
<package pattern="Azure.ResourceManager" />

‎eng/Version.Details.xml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,10 @@
7373
<Uri>https://dev.azure.com/dnceng/internal/_git/dotnet-runtime</Uri>
7474
<Sha>5535e31a712343a63f5d7d796cd874e563e5ac14</Sha>
7575
</Dependency>
76+
<Dependency Name="Microsoft.NET.Runtime.WorkloadTesting.Internal" Version="9.0.0-preview.3.24151.4">
77+
<Uri>https://github.com/dotnet/runtime</Uri>
78+
<Sha>50d6e5d5ffd05dd4034cffd222ea610baedcc326</Sha>
79+
</Dependency>
7680
<Dependency Name="Microsoft.AspNetCore.OpenApi" Version="8.0.2">
7781
<Uri>https://dev.azure.com/dnceng/internal/_git/dotnet-aspnetcore</Uri>
7882
<Sha>da7e9894ce22ef8cc02e5acc56e95a6f8cf8f644</Sha>

‎eng/Versions.props‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,5 +45,6 @@
4545
<MicrosoftEntityFrameworkCoreDesignPackageVersion>8.0.2</MicrosoftEntityFrameworkCoreDesignPackageVersion>
4646
<MicrosoftEntityFrameworkCoreSqlServerPackageVersion>8.0.2</MicrosoftEntityFrameworkCoreSqlServerPackageVersion>
4747
<MicrosoftEntityFrameworkCoreToolsPackageVersion>8.0.2</MicrosoftEntityFrameworkCoreToolsPackageVersion>
48+
<MicrosoftNETRuntimeWorkloadTestingInternalVersion>9.0.0-preview.3.24151.4</MicrosoftNETRuntimeWorkloadTestingInternalVersion>
4849
</PropertyGroup>
4950
</Project>

‎eng/dashboardpack/dashboardpack.csproj‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636
</ItemGroup>
3737

3838
<MakeDir Directories="$(DashboardPublishedArtifactsOutputDir)/$(DashboardRuntime)" />
39-
<ZipDirectory SourceDirectory="$(DotNetOutputPath)Aspire.Dashboard/$(Configuration)/$(TargetFramework)/$(DashboardRuntime)/publish" DestinationFile="$(DashboardPublishedArtifactsOutputDir)/$(DashboardRuntime)/aspire-dashboard-$(DashboardRuntime).zip" />
39+
<ZipDirectory SourceDirectory="$(DotNetOutputPath)Aspire.Dashboard/$(Configuration)/$(TargetFramework)/$(DashboardRuntime)/publish" DestinationFile="$(DashboardPublishedArtifactsOutputDir)/$(DashboardRuntime)/aspire-dashboard-$(DashboardRuntime).zip" Overwrite="true" />
4040

4141
<!-- Throw an error if _PublishItems is empty. -->
4242
<Error Condition="'@(_PublishItems)' == ''" Text="No files were found to pack. Ensure that the project being packed has a publish target defined." />
@@ -48,4 +48,4 @@
4848
<None Include="UnixFilePermissions.xml" Pack="true" PackagePath="data/" Condition=" '$(DashboardPlatformType)' == 'Unix' " />
4949
</ItemGroup>
5050

51-
</Project>
51+
</Project>

‎eng/dcppack/dcppack.csproj‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
<GeneratePackageOnBuild>true</GeneratePackageOnBuild>
1111
<IncludeBuildOutput>false</IncludeBuildOutput>
1212
<PackageOutputPath Condition=" '$(PackageOutputPath)' == '' ">$(ArtifactsShippingPackagesDir)</PackageOutputPath>
13+
<TargetsForTfmSpecificContentInPackage>$(TargetsForTfmSpecificContentInPackage);AddPackageFiles</TargetsForTfmSpecificContentInPackage>
1314
</PropertyGroup>
1415

1516
<PropertyGroup>
@@ -39,8 +40,19 @@
3940

4041
<Target Name="Build" />
4142

43+
<Target Name="AddPackageFiles" Returns="@(TfmSpecificPackageFile)">
44+
<PropertyGroup>
45+
<_DcpNuGetRootPath>$(NuGetPackageRoot)microsoft.developercontrolplane.$(DcpPlatform)/$(MicrosoftDeveloperControlPlanedarwinamd64PackageVersion)/tools</_DcpNuGetRootPath>
46+
</PropertyGroup>
47+
<ItemGroup>
48+
<_DcpFiles Include="$(_DcpNuGetRootPath)\**\*" />
49+
<TfmSpecificPackageFile Include="@(_DcpFiles)" PackagePath="tools/" />
50+
</ItemGroup>
51+
52+
<Error Condition="@(_DcpFiles->Count()) == 0" Text="No dcp files found in : $(_DcpNuGetRootPath)" />
53+
</Target>
54+
4255
<ItemGroup>
43-
<None Include="$(NuGetPackageRoot)Microsoft.DeveloperControlPlane.$(DcpPlatform)/$(MicrosoftDeveloperControlPlanedarwinamd64PackageVersion)/tools/**/*" Pack="true" PackagePath="tools/" />
4456
<None Include="Sdk.props" Pack="true" PackagePath="Sdk/" />
4557
<None Include="Sdk.in.targets" PerformTextReplacement="True" Pack="true" PackagePath="Sdk/" />
4658
<None Include="Aspire.Hosting.Orchestration.props" pack="true" PackagePath="build/$(PackageId).props" />
@@ -52,4 +64,4 @@
5264
<TextReplacementValue Include="RUNTIME" NewValue="$(DcpRuntime)" />
5365
</ItemGroup>
5466

55-
</Project>
67+
</Project>

‎eng/pipelines/templates/BuildAndTest.yml‎

Lines changed: 31 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,26 @@ parameters:
1919
default: false
2020

2121
steps:
22-
- script: ${{ parameters.buildScript }}
23-
-restore -build
24-
-configuration ${{ parameters.buildConfig }}
25-
/p:ArchiveTests=true
26-
/bl:${{ parameters.repoLogPath }}/build.binlog
27-
$(_OfficialBuildIdArgs)
28-
displayName: Build
22+
- ${{ if eq(parameters.isWindows, 'true') }}:
23+
- script: ${{ parameters.buildScript }}
24+
-restore -build
25+
-configuration ${{ parameters.buildConfig }}
26+
/p:ArchiveTests=true
27+
/bl:${{ parameters.repoLogPath }}/build.binlog
28+
$(_OfficialBuildIdArgs)
29+
displayName: Build
30+
- ${{ if ne(parameters.isWindows, 'true') }}:
31+
# temporary: Use -pack for linux as it is required for workload testing
32+
# Issue: https://github.com/dotnet/aspire/issues/2687
33+
- script: ${{ parameters.buildScript }}
34+
-restore -build
35+
-pack
36+
-configuration ${{ parameters.buildConfig }}
37+
/p:ArchiveTests=true
38+
/bl:${{ parameters.repoLogPath }}/build.binlog
39+
$(extraBuildArgs)
40+
$(_OfficialBuildIdArgs)
41+
displayName: Build
2942

3043
- ${{ if ne(parameters.skipTests, 'true') }}:
3144
- script: ${{ parameters.dotnetScript }} dotnet-coverage collect
@@ -42,10 +55,20 @@ steps:
4255
displayName: Publish non-helix TestResults
4356
condition: always()
4457

58+
- ${{ if ne(parameters.isWindows, 'true') }}:
59+
# E2E tests are disabled on windows for now
60+
- script: ${{ parameters.buildScript }}
61+
-p:ContinuousIntegrationBuild=true
62+
-p:Configuration=${{ parameters.buildConfig }}
63+
-p:TriggerInstallWorkloadForTesting=true
64+
-bl:${{ parameters.repoLogPath }}/WorkloadInstallForTesting.binlog
65+
--projects $(Build.SourcesDirectory)/tests/Aspire.EndToEnd.Tests/Aspire.EndToEnd.Tests.csproj
66+
displayName: Install sdk+workload for testing
67+
4568
- template: /eng/pipelines/templates/send-to-helix.yml
4669
parameters:
4770
HelixProjectPath: '$(Build.SourcesDirectory)/tests/send-to-helix.proj'
48-
HelixProjectArguments: /p:RunWithCodeCoverage=true /p:RepoTestResultsPath=${{ parameters.repoTestResultsPath }}
71+
HelixProjectArguments: /p:Configuration=${{ parameters.buildConfig }} /p:RunWithCodeCoverage=true /p:RepoTestResultsPath=${{ parameters.repoTestResultsPath }}
4972

5073
${{ if eq(parameters.isWindows, 'true') }}:
5174
${{ if eq(variables['System.TeamProject'], 'public') }}:

‎eng/testing/.runsettings‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
<?xml version="1.0" encoding="utf-8"?>
22
<RunSettings>
33
<RunConfiguration>
4-
<!-- Timeout in ms, 5 minutes -->
5-
<TestSessionTimeout>300000</TestSessionTimeout>
4+
<!-- Timeout in ms, 10 minutes -->
5+
<TestSessionTimeout>600000</TestSessionTimeout>
66
<!-- Filter out failing (wrong framework, platform, runtime or activeissue) tests -->
77
<TestCaseFilter>category!=failing</TestCaseFilter>
88
</RunConfiguration>

0 commit comments

Comments
 (0)