[tests] EndToEnd - add test scenarios #3113

radical · 2024-03-22T20:33:04Z

Test scenarios allow defining different groupings for tests to be run. This is useful for CI to run the tests in separate helix items.

        TestResourceNames resourcesToInclude = TestScenario switch
        {
            "oracle" => TestResourceNames.oracledatabase,
            "cosmos" => TestResourceNames.cosmos,
            "scenario0" => TestResourceNames.kafka
                              | TestResourceNames.mongodb
                              | TestResourceNames.rabbitmq
                              | TestResourceNames.redis
                              | TestResourceNames.postgres
                              | TestResourceNames.efnpgsql
                              | TestResourceNames.mysql
                              | TestResourceNames.efmysql
                              | TestResourceNames.sqlserver,
            "" or null => TestResourceNames.All,
            _ => throw new ArgumentException($"Unknown test scenario '{TestScenario}'")
        };

Once eforacle is added it can run with the oracle scenario.

The default is to run all the tests.

Also:

Enable cosmos E2E tests on helix
make TestResourceNames into a [Flag] enum so multiple can be expressed at the same time.
And support dumping docker container logs for resources where the resource name does not match the container name prefix. For example efnpgsql vs postgres.

Microsoft Reviewers: Open in CodeFlow

This makes EndToEnd tests a special case of workload tests, IOW, tests that need to use a sdk+workload-installed-from-artifacts. The reason being that these tests need some special logic to queue up multiple helix items to run multiple test scenarios, which differ only in their setup and not the test assembly itself. In follow up PRs with other workload test projects being added, they will default to adding one item but with the sdk+workload.

These tests start up multiple containers and those can be too heavy for the helix machines. Instead, adding test scenarios allows running the tests on multiple helix machines with specific containers. For example: - scenario0 - this is the set of the "light" containers - cosmos - this one takes a lot of time to get ready, thus runs by itself Also, this renames `pomelo` service in the tests to `efmysql`.

In the following code snippet, the old code "retries 60 times", which can total up to 2 mins. And 2 mins is the timeout used by the actual EndtoEnd tests invoking these services. ``` +++ b/tests/testproject/TestProject.IntegrationServiceA/Cosmos/CosmosExtensions.cs @@ -18,7 +18,7 @@ private static async Task<IResult> VerifyCosmosAsync(CosmosClient cosmosClient) var policy = Policy .Handle<HttpRequestException>() // retry 60 times with a 1 second delay between retries - .WaitAndRetryAsync(60, retryAttempt => TimeSpan.FromSeconds(1)); + .WaitAndRetryAsync(20, retryAttempt => TimeSpan.FromSeconds(1)); var db = await policy.ExecuteAsync( async () => (await cosmosClient.CreateDatabaseIfNotExistsAsync("db")).Database); ``` Instead, reduce the number of attempts to a more reasonable number.

radical · 2024-03-29T19:16:49Z

tests/testproject/TestProject.IntegrationServiceA/SqlServer/SqlServerExtensions.cs

@@ -18,7 +18,7 @@ private static async Task<IResult> VerifySqlServerAsync(SqlConnection connection
            var policy = Policy
                .Handle<SqlException>()
                // retry 60 times with a 1 second delay between retries
-                .WaitAndRetryAsync(60, retryAttempt => TimeSpan.FromSeconds(1));
+                .WaitAndRetryAsync(10, retryAttempt => TimeSpan.FromSeconds(1));


@eerhardt @sebastienros Is there a better way to express a "overall timeout" for the attempts here, so it all completes under n mins? Otherwise, this keeps retrying and the caller (test) fails because of it's own timeout (2mins).

eerhardt · 2024-04-03T16:15:14Z

Directory.Build.props

@@ -45,8 +45,9 @@
    <BuildArch Condition=" '$(BuildArch)' == '' ">amd64</BuildArch>
    <DcpDir>$(NuGetPackageRoot)microsoft.developercontrolplane.$(BuildOs)-$(BuildArch)/$(MicrosoftDeveloperControlPlanedarwinamd64PackageVersion)/tools/</DcpDir>

-    <TestArchiveTestsDir>$([MSBuild]::NormalizeDirectory($(ArtifactsDir), 'helix', 'tests'))</TestArchiveTestsDir>
+    <TestArchiveTestsDir Condition="'$(TestArchiveTestsDir)' == ''">$([MSBuild]::NormalizeDirectory($(ArtifactsDir), 'helix', 'tests'))</TestArchiveTestsDir>


Can these "Test*" properties be moved to the tests\Directory.Build.props|targets? That way they don't clutter up the root.

The root props and targets should only be for things that apply across all projects in the repo.

eerhardt · 2024-04-03T16:16:16Z

Directory.Build.targets

@@ -5,7 +5,7 @@
    <ReadMeExists Condition="Exists('$(ReadMePath)')">true</ReadMeExists>
    <PackageReadmeFile Condition="'$(PackageReadmeFile)' == '' And '$(ReadMeExists)' == 'true'">README.md</PackageReadmeFile>

-    <TestArchiveTestsDir Condition="'$(IsWorkloadTestProject)' == 'true'">$(TestArchiveTestsDirForWorkloadTests)</TestArchiveTestsDir>
+    <TestArchiveTestsDir Condition="'$(TestArchiveTestsDir)' == '' and '$(IsWorkloadTestProject)' == 'true'">$(TestArchiveTestsDirForWorkloadTests)</TestArchiveTestsDir>


This is weird that this property is set in both the .props and the .targets. Why both?

Can this condition ever be true since it is already defaulted in the .props?

eerhardt · 2024-04-03T16:18:39Z

tests/send-to-helix-workload-tests.proj

        <Command>$(_TestRunCommand)</Command>
        <Timeout>$(_WorkItemTimeoutForWorkloadTests)</Timeout>

        <!-- Download results file so coverage files can be extracted -->
-        <DownloadFilesFromResults>logs/%(FileName).cobertura.xml</DownloadFilesFromResults>
+        <!-- FIXME: does sharing the TEST_NAME affect the reports? Yes, TestNameEnvVar - gets used for these reports. Make sure they end up in different directories! -->


What's the plan to address this?

this can be removed now as I had checked the code-coverage reports and they were unaffected.

eerhardt · 2024-04-03T16:19:47Z

tests/send-to-helix-workload-tests.proj

@@ -51,18 +64,23 @@
           Text="Could not find dotnet-coverage tool. %24(_DotNetCoverageToolPath)=$(_DotNetCoverageToolPath)" />

    <ItemGroup>
+      <_E2ETestScenarios Include="scenario0" />


Can we give this a better name than scenario0?

eerhardt · 2024-04-03T16:21:03Z

tests/Aspire.EndToEnd.Tests/IntegrationServicesTests.cs

-    [InlineData(TestResourceNames.oracledatabase)]
-    public Task VerifyComponentWorksDisabledOnCI(TestResourceNames resourceName)
+    [Theory]
+    [Trait("scenario", "sqlserver")]


Why are we splitting SqlServer out?

eerhardt · 2024-04-03T16:29:26Z

tests/testproject/TestProject.IntegrationServiceA/Cosmos/CosmosExtensions.cs

@@ -18,7 +18,7 @@ private static async Task<IResult> VerifyCosmosAsync(CosmosClient cosmosClient)
            var policy = Policy
                .Handle<HttpRequestException>()
                // retry 60 times with a 1 second delay between retries
-                .WaitAndRetryAsync(60, retryAttempt => TimeSpan.FromSeconds(1));
+                .WaitAndRetryAsync(20, retryAttempt => TimeSpan.FromSeconds(1));


The comment is out of date.

Also, why are we lowering this retry time?

eerhardt · 2024-04-03T16:51:29Z

tests/send-to-helix-workload-tests.proj

@@ -2,6 +2,9 @@



Is this file really a .proj file? Since it is imported by the send-to-helix.proj, isn't it more of a .targets file?

…at SNS Subscription will work. (dotnet#3333)

…et#3320) * Fix ConfigurationSchemaGenerator to use correct TimeSpan format 'duration' isn't an acceptable description of how to represent a TimeSpan. Instead use a custom regular expression to represent TimeSpan formats. * Add tests for the timespan regex. * Fix outdated JSON config schema tests * Fix more tests

The new versions have specific builds for the .NETCoreApp TFM, which fixes some dependency issues like not having the 8.0.0 version of Microsoft.Bcl.AsyncInterfaces. See Xabaril/AspNetCore.Diagnostics.HealthChecks#2180.

…: Build ID 2420391 (dotnet#3335) * Localized file check-in by OneLocBuild Task: Build definition ID 1309: Build ID 2419838 * Localized file check-in by OneLocBuild Task: Build definition ID 1309: Build ID 2419838 * Localized file check-in by OneLocBuild Task: Build definition ID 1309: Build ID 2419838 * Localized file check-in by OneLocBuild Task: Build definition ID 1309: Build ID 2419838 * Localized file check-in by OneLocBuild Task: Build definition ID 1309: Build ID 2419960

…isplay them (dotnet#3235) Change the AppHost to only subscribe to DCP's logs when there is a subscriber for the logs. Fix dotnet#2789 * Show logs * Remove the "StreamingLogger" because that causes subsequent subscribers to not see what was already written. Fix duplicate logs issue by clearing out the backlog when the last subscriber leaves. * Fix a concurrency issue with the backlog. Ensure the backlog is only snap shotted after subscribing to the log. Fix existing tests for new functionality and add an additional test. * PR feedback Only clear the backlog on containers and executables. * PR feedback. Move lock out of async iterator. * Clean up code. - Remove count and instead check if the delegate is null to indicate whether there are subscribers or not. - Remove the separate IAsyncEnumerable classes and just use `async IAsyncEnumerable` methods. * Fix a race at startup when logs aren't available. Employ a 2nd loop that listens for both "has subscribers" and "logs available". * Simplify ResourceNotificationService.WatchAsync. * Fix test build * Address PR feedback - Set SingleReader=true on the logInformationChannel. - Add comment and assert that LogsAvailable can only turn true and can't go back to false. --------- Co-authored-by: David Fowler <davidfowl@gmail.com>

* Improve service address allocation Should fix dotnet#3265

* Added ability to resolve target port - Change how we process and expose target ports. This change makes it possible to avoid using any internals of DCP to access and resolve target ports from any resource. - More launch profile processing to the project resource defaults. - Remove logic in the manifest writing that wrote a special env variable. This should just work now like everything else. - Exposed TargetPort and Exists on EndpointReference. - Use this new target property with dapr. - Added a ReferenceExpressionBuilder which makes it possible to dynamically build a ReferenceExpression. - Endpoints and env are evaulated earlier now so change remove tests that add default launch profile on real projects without allocating those endpoints. * Remove more uses of TestProgram * Fix tests

* Tweak Bicep string name. * Fix up test.

* Update OpenTelemetry packages to 1.8.0 * Update template versions

…ices

.. and explicitly set the `$(TestArchiveTestsDir)` to `$(TestArchiveTestsDirForEndToEndTests)` in `EndToEnd.Tests.csproj`.

…llow up PR

radical · 2024-04-03T19:13:17Z

Hopefully, I didn't break the tests!

tests/Aspire.EndToEnd.Tests/IntegrationServicesFixture.cs

eerhardt · 2024-04-03T20:40:09Z

tests/send-to-helix-workload-tests.targets

@@ -51,18 +64,21 @@
           Text="Could not find dotnet-coverage tool. %24(_DotNetCoverageToolPath)=$(_DotNetCoverageToolPath)" />

    <ItemGroup>
+      <_E2ETestScenarios Include="basicservices" />
+      <_E2ETestScenarios Include="cosmos" />


? I thought we didn't run this in CI.

or is the intention to start running it now?

Yes, having it run separately it works fine.

eerhardt · 2024-04-03T20:40:56Z

tests/send-to-helix-workload-tests.targets

        <Command>$(_TestRunCommand)</Command>
        <Timeout>$(_WorkItemTimeoutForWorkloadTests)</Timeout>

        <!-- Download results file so coverage files can be extracted -->
-        <DownloadFilesFromResults>logs/%(FileName).cobertura.xml</DownloadFilesFromResults>
+        <DownloadFilesFromResults>logs/Aspire.EndToEnd.Tests.cobertura.xml</DownloadFilesFromResults>


Isn't this name going to conflict between multiple _E2ETestScenarios ?

hmph yeah:

Copying /mnt/vss/_work/1/s/artifacts/helixresults/cc085e87-12fa-4085-8946-dc76e4f0f4c1/Aspire.EndToEnd.Tests-cosmos/logs/Aspire.EndToEnd.Tests.cobertura.xml to /mnt/vss/_work/1/s/artifacts/CodeCoverage/Aspire.EndToEnd.Tests.cobertura.xml File /mnt/vss/_work/1/s/artifacts/helixresults/cc085e87-12fa-4085-8946-dc76e4f0f4c1/Aspire.EndToEnd.Tests-scenario0/logs/Aspire.EndToEnd.Tests.cobertura.xml already exist at /mnt/vss/_work/1/s/artifacts/CodeCoverage/Aspire.EndToEnd.Tests.cobertura.xml File /mnt/vss/_work/1/s/artifacts/helixresults/cc085e87-12fa-4085-8946-dc76e4f0f4c1/Aspire.EndToEnd.Tests-sqlserver/logs/Aspire.EndToEnd.Tests.cobertura.xml already exist at /mnt/vss/_work/1/s/artifacts/CodeCoverage/Aspire.EndToEnd.Tests.cobertura.xml

eerhardt

Just a couple minor suggestions.

Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>

…ween them

joperezr · 2024-04-03T23:16:46Z

Thanks for this @radical . I'm assuming we want to backport it for P6 so we get coverage in release branches too?

joperezr · 2024-04-03T23:50:07Z

/backport to release/8.0

github-actions · 2024-04-03T23:50:17Z

Started backporting to release/8.0: https://github.com/dotnet/aspire/actions/runs/8547015795

github-actions · 2024-04-03T23:50:21Z

@joperezr backporting to release/8.0 failed, the patch most likely resulted in conflicts:

$ git am --3way --ignore-whitespace --keep-non-patch changes.patch

Applying: [tests] Make E2E tests a special case of workload tests
Applying: [tests] Add test scenarios to EndToEnd tests
Applying: [tests] Fix retries in the testproject services
Applying: cleanup
Applying: Cleanup
Applying: Remove obsolete APIs. (#3329)
Using index info to reconstruct a base tree...
M	src/Aspire.Hosting.Azure.Storage/AzureStorageExtensions.cs
M	src/Aspire.Hosting.Azure/AzureBicepResource.cs
M	src/Aspire.Hosting.Oracle/OracleDatabaseBuilderExtensions.cs
M	src/Aspire.Hosting/ApplicationModel/EnvironmentCallbackContext.cs
M	src/Aspire.Hosting/ExecutableResourceBuilderExtensions.cs
M	src/Aspire.Hosting/ProjectResourceBuilderExtensions.cs
M	src/Components/Aspire.Azure.AI.OpenAI/AspireAzureOpenAIExtensions.cs
M	src/Components/Aspire.Azure.Data.Tables/AspireTablesExtensions.cs
M	src/Components/Aspire.Azure.Messaging.ServiceBus/AspireServiceBusExtensions.cs
M	src/Components/Aspire.Azure.Search.Documents/AspireAzureSearchExtensions.cs
M	src/Components/Aspire.Azure.Security.KeyVault/AspireKeyVaultExtensions.cs
M	src/Components/Aspire.Azure.Storage.Blobs/AspireBlobStorageExtensions.cs
M	src/Components/Aspire.Azure.Storage.Queues/AspireQueueStorageExtensions.cs
M	src/Components/Aspire.Microsoft.Azure.Cosmos/AspireAzureCosmosDBExtensions.cs
M	src/Components/Aspire.RabbitMQ.Client/AspireRabbitMQExtensions.cs
M	src/Components/Aspire.StackExchange.Redis/AspireRedisExtensions.cs
M	src/Microsoft.Extensions.ServiceDiscovery/ServiceDiscoveryHttpClientBuilderExtensions.cs
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
Applying: In AWS Playground AppHost, add QueuePolicy to ChatMessagesQueue so that SNS Subscription will work. (#3333)
Applying: Remove Azure SDK feed from nuget.config (#3327)
Using index info to reconstruct a base tree...
M	NuGet.config
Falling back to patching base and 3-way merge...
No changes -- Patch already applied.
Applying: Fix ConfigurationSchemaGenerator to use correct TimeSpan format (#3320)
Applying: Update AspNetCore.HealthChecks.* to latest versions (#3338)
Using index info to reconstruct a base tree...
M	Directory.Packages.props
Falling back to patching base and 3-way merge...
Auto-merging Directory.Packages.props
No changes -- Patch already applied.
Applying: Localized file check-in by OneLocBuild Task: Build definition ID 1309: Build ID 2420391 (#3335)
Applying: Dashboard should request log streams only when the user gestures to display them (#3235)
Applying: Publish build assets in the build stage (#3334)
Applying: Improve service address allocation (#3294)
.git/rebase-apply/patch:108: trailing whitespace.
            
warning: 1 line adds whitespace errors.
Using index info to reconstruct a base tree...
M	src/Aspire.Hosting/Dcp/ApplicationExecutor.cs
Falling back to patching base and 3-way merge...
Auto-merging src/Aspire.Hosting/Dcp/ApplicationExecutor.cs
CONFLICT (content): Merge conflict in src/Aspire.Hosting/Dcp/ApplicationExecutor.cs
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0014 Improve service address allocation (#3294)
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

github-actions · 2024-04-03T23:50:22Z

@joperezr an error occurred while backporting to release/8.0, please check the run log for details!

Error: git am failed, most likely due to a merge conflict.

Test scenarios allow defining different groupings for tests to be run. This is useful for CI to run the tests in separate helix items. ```csharp TestResourceNames resourcesToInclude = TestScenario switch { "oracle" => TestResourceNames.oracledatabase, "cosmos" => TestResourceNames.cosmos, "scenario0" => TestResourceNames.kafka | TestResourceNames.mongodb | TestResourceNames.rabbitmq | TestResourceNames.redis | TestResourceNames.postgres | TestResourceNames.efnpgsql | TestResourceNames.mysql | TestResourceNames.efmysql | TestResourceNames.sqlserver, "" or null => TestResourceNames.All, _ => throw new ArgumentException($"Unknown test scenario '{TestScenario}'") }; ``` Once `eforacle` is added it can run with the `oracle` scenario. The default is to run all the tests. Also: - Enable `cosmos` E2E tests on helix - make `TestResourceNames` into a `[Flag]` enum so multiple can be expressed at the same time. - And support dumping docker container logs for resources where the resource name does not match the container name prefix. For example `efnpgsql` vs `postgres`.

Test scenarios allow defining different groupings for tests to be run. This is useful for CI to run the tests in separate helix items. ```csharp TestResourceNames resourcesToInclude = TestScenario switch { "oracle" => TestResourceNames.oracledatabase, "cosmos" => TestResourceNames.cosmos, "scenario0" => TestResourceNames.kafka | TestResourceNames.mongodb | TestResourceNames.rabbitmq | TestResourceNames.redis | TestResourceNames.postgres | TestResourceNames.efnpgsql | TestResourceNames.mysql | TestResourceNames.efmysql | TestResourceNames.sqlserver, "" or null => TestResourceNames.All, _ => throw new ArgumentException($"Unknown test scenario '{TestScenario}'") }; ``` Once `eforacle` is added it can run with the `oracle` scenario. The default is to run all the tests. Also: - Enable `cosmos` E2E tests on helix - make `TestResourceNames` into a `[Flag]` enum so multiple can be expressed at the same time. - And support dumping docker container logs for resources where the resource name does not match the container name prefix. For example `efnpgsql` vs `postgres`. Co-authored-by: Ankit Jain <radical@gmail.com>

ghost added the area-app-model Issues pertaining to the APIs in Aspire.Hosting, e.g. DistributedApplication label Mar 22, 2024

radical added 3 commits March 29, 2024 15:13

radical force-pushed the e2e-scenarios-staging branch from 658bb5b to be2f7ee Compare March 29, 2024 19:14

radical commented Mar 29, 2024

View reviewed changes

cleanup

12763ed

radical marked this pull request as ready for review March 29, 2024 19:21

radical requested review from eerhardt, mitchdenny and joperezr and removed request for mitchdenny March 29, 2024 19:21

radical added the testing ☑️ label Apr 1, 2024

radical added 2 commits April 1, 2024 22:47

Merge branch 'dotnet:main' into e2e-scenarios-staging

8c4b070

Cleanup

2de4678

eerhardt reviewed Apr 3, 2024

View reviewed changes

mitchdenny and others added 14 commits April 3, 2024 14:52

Remove obsolete APIs. (dotnet#3329)

cbcc9f9

In AWS Playground AppHost, add QueuePolicy to ChatMessagesQueue so th…

63ad90c

…at SNS Subscription will work. (dotnet#3333)

Remove Azure SDK feed from nuget.config (dotnet#3327)

be60a0f

Update AspNetCore.HealthChecks.* to latest versions (dotnet#3338)

fa9ffe5

The new versions have specific builds for the .NETCoreApp TFM, which fixes some dependency issues like not having the 8.0.0 version of Microsoft.Bcl.AsyncInterfaces. See Xabaril/AspNetCore.Diagnostics.HealthChecks#2180.

Publish build assets in the build stage (dotnet#3334)

727d676

Improve service address allocation (dotnet#3294)

632beff

* Improve service address allocation Should fix dotnet#3265

Add token auth to dashboard frontend (dotnet#3213)

e189200

Tweak Bicep string name. (dotnet#3331)

b940c0c

* Tweak Bicep string name. * Fix up test.

Improve writing dashboard startup config failure messages (dotnet#3243)

727ec02

Update OpenTelemetry packages to 1.8.0 (dotnet#3351)

3d74b7f

* Update OpenTelemetry packages to 1.8.0 * Update template versions

radical added 5 commits April 3, 2024 14:57

rename send-to-helix-workload-tests.proj to .targets

fdcadc9

rename scenario0 to basicservices, and merge sqlserver with basicserv…

c4bf749

…ices

Move $(Test*) to test/Directory.Build.props

2ca62ae

.. and explicitly set the `$(TestArchiveTestsDir)` to `$(TestArchiveTestsDirForEndToEndTests)` in `EndToEnd.Tests.csproj`.

Revert timeout changes from services. this will get addressed in a fo…

ce99dd3

…llow up PR

remove resolved fixme

d87cedb

radical added 2 commits April 3, 2024 15:13

Merge remote-tracking branch 'origin/main' into e2e-scenarios-staging

ed07e88

fix scenario0 everywhere

f1b7360

eerhardt reviewed Apr 3, 2024

View reviewed changes

tests/Aspire.EndToEnd.Tests/IntegrationServicesFixture.cs Outdated Show resolved Hide resolved

eerhardt reviewed Apr 3, 2024

View reviewed changes

tests/Aspire.EndToEnd.Tests/IntegrationServicesFixture.cs Outdated Show resolved Hide resolved

eerhardt reviewed Apr 3, 2024

View reviewed changes

eerhardt approved these changes Apr 3, 2024

View reviewed changes

radical and others added 4 commits April 3, 2024 16:49

Update tests/Aspire.EndToEnd.Tests/IntegrationServicesFixture.cs

a6f2510

Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>

Update tests/Aspire.EndToEnd.Tests/IntegrationServicesFixture.cs

3abcac1

Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>

Use scenario name in cobertura.xml for e2e tests to differentiate bet…

cb2de39

…ween them

fix reference to renamed property

931996f

radical merged commit a9175de into dotnet:main Apr 3, 2024

radical deleted the e2e-scenarios-staging branch April 3, 2024 22:54

joperezr mentioned this pull request Apr 3, 2024

[release/8.0] [tests] EndToEnd - add test scenarios (#3113) #3377

Merged

radical mentioned this pull request Apr 4, 2024

[tests] Follow-up work for E2E testing with workloads #2687

Closed

9 tasks

danmoseley mentioned this pull request Apr 12, 2024

azure.identity.bump #3659

Closed

github-actions bot locked and limited conversation to collaborators May 4, 2024

[tests] EndToEnd - add test scenarios #3113

[tests] EndToEnd - add test scenarios #3113

Uh oh!

Conversation

radical commented Mar 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Microsoft Reviewers: Open in CodeFlow

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

radical commented Apr 3, 2024

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eerhardt left a comment

Choose a reason for hiding this comment

Uh oh!

joperezr commented Apr 3, 2024

Uh oh!

joperezr commented Apr 3, 2024

Uh oh!

github-actions bot commented Apr 3, 2024

Uh oh!

github-actions bot commented Apr 3, 2024

Uh oh!

github-actions bot commented Apr 3, 2024

Uh oh!

Uh oh!

radical commented Mar 22, 2024 •

edited

Loading