Fix SupportedRuntimeIdentifiers substitution in Directory.Build.targets.in. #48022

tmds · 2023-05-02T14:16:29Z

#47247 enabled AOT template tests.
From the comments on the PR, it seems the tests are known to fail on Linux.

This adds attributes to skip those tests on Linux, like when running on a Linux developer machine, or our internal CI machines.

@JamesNK @dougbu @eerhardt ptal.

ghost · 2023-05-02T14:16:37Z

Thanks for your PR, @tmds. Someone from the team will get assigned to your PR shortly and we'll get it reviewed.

eerhardt · 2023-05-02T14:39:13Z

I was under the assumption that the tests only fail in Helix because we only run the Helix tests against the Windows-built runtime pack / shared framework. They should pass in a local build.

@tmds - what error are you seeing when they run?

cc @mitchdenny @wtgodbe

tmds · 2023-05-02T14:51:02Z

I have not run them on my machine, on our Linux CI server I see:

      <test name="Templates.Test.GrpcTemplateTest.GrpcTemplateNativeAot" type="Templates.Test.GrpcTemplateTest" method="GrpcTemplateNativeAot" time="94.2551574" result="Fail">
        <output><![CDATA[| [0.001s] TestLifetime Information: Starting test GrpcTemplateNativeAot at 2023-05-01T23:40:29
| [0.002s] Templates.Test.GrpcTemplateTest Information: ==> /home/tester/aspnetcore/.dotnet/dotnet new grpc  --debug:disable-sdk-templates --debug:custom-hive "/home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/Hives/b473dcd1-4727-479d-acb1-25d79f0e2d27/.templateengine" --aot -o /home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman [/home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/]
| [0.948s] Templates.Test.GrpcTemplateTest Information: The template "ASP.NET Core gRPC Service" was created successfully.
| [0.949s] Templates.Test.GrpcTemplateTest Information: 
| [0.949s] Templates.Test.GrpcTemplateTest Information: Processing post-creation actions...
| [0.950s] Templates.Test.GrpcTemplateTest Information: Restoring /home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj:
| [1.996s] Templates.Test.GrpcTemplateTest Information:   Determining projects to restore...
| [56.579s] Templates.Test.GrpcTemplateTest Information:   Restored /home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj (in 53.68 sec).
| [56.624s] Templates.Test.GrpcTemplateTest Information: Restore succeeded.
| [56.624s] Templates.Test.GrpcTemplateTest Information: 
| [56.626s] Templates.Test.GrpcTemplateTest Information: 
| [56.627s] Templates.Test.GrpcTemplateTest Information: [ERROR] Failed to check update for Microsoft.DotNet.Web.ProjectTemplates.8.0::8.0.0-ci: Failed to check the update for the package Microsoft.DotNet.Web.ProjectTemplates.8.0::8.0.0-ci.
| [56.627s] Templates.Test.GrpcTemplateTest Information: [ERROR] Details: Value cannot be null. (Parameter 'releasePrefix')..
| [56.627s] Templates.Test.GrpcTemplateTest Information: [ERROR] 
| [56.649s] Templates.Test.GrpcTemplateTest Information: Process exited.
| [56.650s] Templates.Test.GrpcTemplateTest Information: Publishing ASP.NET Core application...
| [56.650s] Templates.Test.GrpcTemplateTest Information: ==> /home/tester/aspnetcore/.dotnet/dotnet publish  -c Release /bl  [/home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman]
| [56.879s] Templates.Test.GrpcTemplateTest Information: MSBuild version 17.7.0-preview-23225-01+6300d22b2 for .NET
| [58.085s] Templates.Test.GrpcTemplateTest Information:   Determining projects to restore...
| [93.857s] Templates.Test.GrpcTemplateTest Information:   Restored /home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj (in 34.97 sec).
| [93.996s] Templates.Test.GrpcTemplateTest Information: /home/tester/aspnetcore/.dotnet/sdk/8.0.100-preview.5.23226.2/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(287,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj]
| [94.150s] Templates.Test.GrpcTemplateTest Information: /home/tester/aspnetcore/.dotnet/sdk/8.0.100-preview.5.23226.2/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(478,5): error NETSDK1082: There was no runtime pack for Microsoft.AspNetCore.App available for the specified RuntimeIdentifier 'linux-x64'. [/home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj]
| [94.252s] Templates.Test.GrpcTemplateTest Information: Process exited.
| [94.257s] Templates.Test.GrpcTemplateTest Error: Test threw an exception.
| Xunit.Sdk.TrueException: Project new grpc  --aot failed to publish. Exit code 1.
| /home/tester/aspnetcore/.dotnet/dotnet publish  -c Release /bl \nStdErr: \nStdOut: MSBuild version 17.7.0-preview-23225-01+6300d22b2 for .NET
|   Determining projects to restore...
|   Restored /home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj (in 34.97 sec).
| /home/tester/aspnetcore/.dotnet/sdk/8.0.100-preview.5.23226.2/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(287,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj]
| /home/tester/aspnetcore/.dotnet/sdk/8.0.100-preview.5.23226.2/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(478,5): error NETSDK1082: There was no runtime pack for Microsoft.AspNetCore.App available for the specified RuntimeIdentifier 'linux-x64'. [/home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj]
| Expected: True
| Actual:   False
|    at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
|    at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 116
|    at Templates.Test.Helpers.Project.RunDotNetPublishAsync(IDictionary`2 packageOptions, String additionalArgs, Boolean noRestore) in /_/src/ProjectTemplates/Shared/Project.cs:line 152
|    at Templates.Test.GrpcTemplateTest.GrpcTemplateCore(String[] args) in /_/src/ProjectTemplates/test/Templates.Tests/GrpcTemplateTest.cs:line 91
|    at Templates.Test.GrpcTemplateTest.GrpcTemplateNativeAot() in /_/src/ProjectTemplates/test/Templates.Tests/GrpcTemplateTest.cs:line 52
|    at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_0.<<InvokeTestMethodAsync>b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 264
| --- End of stack trace from previous location ---
|    at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
|    at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90
| [94.259s] TestLifetime Information: Finished test GrpcTemplateNativeAot in 94.2583206s
]]></output>
        <failure exception-type="Xunit.Sdk.TrueException">
          <message><![CDATA[Project new grpc  --aot failed to publish. Exit code 1.\n/home/tester/aspnetcore/.dotnet/dotnet publish  -c Release /bl \\nStdErr: \\nStdOut: MSBuild version 17.7.0-preview-23225-01+6300d22b2 for .NET\n  Determining projects to restore...\n  Restored /home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj (in 34.97 sec).\n/home/tester/aspnetcore/.dotnet/sdk/8.0.100-preview.5.23226.2/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(287,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj]\n/home/tester/aspnetcore/.dotnet/sdk/8.0.100-preview.5.23226.2/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(478,5): error NETSDK1082: There was no runtime pack for Microsoft.AspNetCore.App available for the specified RuntimeIdentifier 'linux-x64'. [/home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj]\n\nExpected: True\nActual:   False]]></message>
          <stack-trace><![CDATA[   at Templates.Test.Helpers.Project.RunDotNetPublishAsync(IDictionary`2 packageOptions, String additionalArgs, Boolean noRestore) in /_/src/ProjectTemplates/Shared/Project.cs:line 152
   at Templates.Test.GrpcTemplateTest.GrpcTemplateCore(String[] args) in /_/src/ProjectTemplates/test/Templates.Tests/GrpcTemplateTest.cs:line 91
   at Templates.Test.GrpcTemplateTest.GrpcTemplateNativeAot() in /_/src/ProjectTemplates/test/Templates.Tests/GrpcTemplateTest.cs:line 52
--- End of stack trace from previous location ---]]></stack-trace>
        </failure>
      </test>

The CI job runs:

source activate.sh
eng/build.sh --ci --arch x64 -c Release /p:IsPlaywrightAvailable=false --restore --build --pack
eng/build.sh --ci --arch x64 -c Release /p:IsPlaywrightAvailable=false --test

eerhardt · 2023-05-02T14:54:53Z

@wtgodbe - any thoughts on why the runtime pack for linux-x64 wouldn't be available after a build.sh --build --pack call?

wtgodbe · 2023-05-02T16:17:47Z

@wtgodbe - any thoughts on why the runtime pack for linux-x64 wouldn't be available after a build.sh --build --pack call?

It would be, my guess is the tests aren't able to access it. Template tests can be weird since they run from a temp dir, so they don't get access to all of our .props/.targets files (though we give them access to enough that I'd expect this scenario to work, so clearly something is going wrong). Maybe these work in Helix because we explicitly lay things out such that they'll have access to the built SharedFx, but local builds would always fail. Do they pass locally on Windows?

The easiest way to diagnose this would be if we could look at the generated .csproj that runs the test in a local build (e.g. /home/tester/aspnetcore/src/ProjectTemplates/test/Templates.Tests/bin/Release/net8.0/TestTemplates/AspNet.Pgpzatoryman/AspNet.Pgpzatoryman.csproj), and the folder(s) it lives in (namely the Directory.Build.Props and Directory.Build.Targets files there)

Tratcher · 2023-05-02T18:42:31Z

The tests are failing on windows too.
https://dev.azure.com/dnceng-public/public/_build/results?buildId=259113&view=ms.vss-test-web.build-test-results-tab&runId=5066154&resultId=123336&paneView=debug

[1.877s] [Templates.Test.ApiTemplateTest] [Information] [ERROR] Failed to check update for Microsoft.DotNet.Web.ProjectTemplates.8.0::8.0.0-ci: Failed to check the update for the package Microsoft.DotNet.Web.ProjectTemplates.8.0::8.0.0-ci.
[1.877s] [Templates.Test.ApiTemplateTest] [Information] [ERROR] Details: Value cannot be null. (Parameter 'releasePrefix')..
[1.877s] [Templates.Test.ApiTemplateTest] [Information] [ERROR] 
[1.904s] [Templates.Test.ApiTemplateTest] [Information] Process exited.
[1.906s] [Templates.Test.ApiTemplateTest] [Information] Publishing ASP.NET Core application...
[1.906s] [Templates.Test.ApiTemplateTest] [Information] ==> C:\h\w\BE500AC4\p\dotnet-cli\dotnet.exe publish  -c Release /bl  [C:\h\w\BE500AC4\w\AE3408ED\e\Templates\BaseFolder\AspNet.Ibjtal4eg0s1]
[2.135s] [Templates.Test.ApiTemplateTest] [Information] MSBuild version 17.7.0-preview-23225-01+6300d22b2 for .NET
[2.810s] [Templates.Test.ApiTemplateTest] [Information]   Determining projects to restore...
[3.430s] [Templates.Test.ApiTemplateTest] [Information]   Restored C:\h\w\BE500AC4\w\AE3408ED\e\Templates\BaseFolder\AspNet.Ibjtal4eg0s1\AspNet.Ibjtal4eg0s1.csproj (in 274 ms).
[3.546s] [Templates.Test.ApiTemplateTest] [Information] C:\h\w\BE500AC4\p\dotnet-cli\sdk\8.0.100-preview.5.23226.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(287,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [C:\h\w\BE500AC4\w\AE3408ED\e\Templates\BaseFolder\AspNet.Ibjtal4eg0s1\AspNet.Ibjtal4eg0s1.csproj]
[3.673s] [Templates.Test.ApiTemplateTest] [Information] C:\h\w\BE500AC4\p\dotnet-cli\sdk\8.0.100-preview.5.23226.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(478,5): error NETSDK1082: There was no runtime pack for Microsoft.AspNetCore.App available for the specified RuntimeIdentifier 'win-arm64'. [C:\h\w\BE500AC4\w\AE3408ED\e\Templates\BaseFolder\AspNet.Ibjtal4eg0s1\AspNet.Ibjtal4eg0s1.csproj]
[3.749s] [Templates.Test.ApiTemplateTest] [Information] Process exited.
[3.754s] [Templates.Test.ApiTemplateTest] [Error] Test threw an exception.
Xunit.Sdk.TrueException: Project new api  --aot failed to publish. Exit code 1.

Tratcher · 2023-05-02T18:44:50Z

Also:

[8.216s] [Templates.Test.ApiTemplateTest] [Information] C:\h\w\AF280944\w\A63608BE\e\nugetRestore\microsoft.dotnet.ilcompiler\8.0.0-preview.4.23225.9\build\Microsoft.NETCore.Native.Windows.targets(122,5): error : Platform linker not found. To fix this problem, download and install Visual Studio 2022 from http://visualstudio.com. Make sure to install the Desktop Development for C++ workload. For ARM64 development also install C++ ARM64 build tools. [C:\h\w\AF280944\w\A63608BE\e\Templates\BaseFolder\AspNet.F1hmjqvwjqmg\AspNet.F1hmjqvwjqmg.csproj]
[8.265s] [Templates.Test.ApiTemplateTest] [Information] Process exited.
[8.268s] [Templates.Test.ApiTemplateTest] [Error] Test threw an exception.
Xunit.Sdk.TrueException: Project new api  --aot failed to publish. Exit code 1.

eerhardt · 2023-05-02T18:51:52Z

error : Platform linker not found. To fix this problem, download and install Visual Studio 2022 from http://visualstudio.com. Make sure to install the Desktop Development for C++ workload.

In order to run the tests, you need C++ on the machine.

Tratcher · 2023-05-02T19:04:14Z

That's coming from the CI, so our agents aren't correctly configured?

eerhardt · 2023-05-02T19:13:47Z

That's coming from the CI, so our agents aren't correctly configured?

Maybe? I know that in #47247 @mitchdenny needed to use specific Helix machines to get the tests to pass.

wtgodbe · 2023-05-02T21:40:57Z

Yeah, these tests were only able to run on a specific Helix queue that had the right VS components installed.

As for the SharedFx issue, do these tests use the shared template test infra like https://github.com/dotnet/aspnetcore/blob/main/src/ProjectTemplates/TestInfrastructure/Directory.Build.props.in? That should already be set up to get the tests restoring the just-built SharedFx, as long as it's already built. Seeing the temp .csproj & its associated Dir.Build.Props/Targets files would give us the info we need.

tmds · 2023-05-03T08:13:47Z

@wtgodbe I'm trying to understand if this is an issue specific to our CI server, or if these tests aren't passing anywhere but the specific Helix queues?

wtgodbe · 2023-05-03T16:04:15Z

I'm trying to understand if this is an issue specific to our CI server, or if these tests aren't passing anywhere but the specific Helix queues?

My understanding is that they should pass on any machine w/ the right C++ tools, but it seems like we're learning that's not that case. That being said, I don't know much about these specific tests, @mitchdenny has more context than I do there (e.g. on whether or not they can pass locally).

mitchdenny · 2023-05-04T01:33:42Z

Not that much more context ;) My goal was to just get some coverage of the templates and we accepted them running on just Windows as a good starting point. It would be great if we could get them running on Linux also but we couldn't get it working (I think it was an error related to a runtime targetting pack).

RussKie · 2023-05-04T05:56:27Z

src/ProjectTemplates/test/Templates.Tests/ApiTemplateTest.cs

@JamesNK @mitchdenny I think this list has to be extended to include all other queues where the test fail.

Ditto for the other three tests.

tmds · 2023-05-04T14:33:14Z

I debugged the issue and the problem is with the newlines/whitespace in SupportedRuntimeIdentifiers.
I assumed this only worked with the rid on the first line (win-x64).
When I remove the newlines (I've updated the PR for that), the aot tests pass on my Linux machine.

An alternative fix could be to do something to the SupportedRuntimeIdentifiers here:

aspnetcore/eng/tools/GenerateFiles/GenerateFiles.csproj

Line 29 in 576c75c

SupportedRuntimeIdentifiers=$(SupportedRuntimeIdentifiers.Trim());

RussKie · 2023-05-05T00:29:25Z

An alternative fix could be to do something to the SupportedRuntimeIdentifiers here:

aspnetcore/eng/tools/GenerateFiles/GenerateFiles.csproj

Line 29 in 576c75c

SupportedRuntimeIdentifiers=$(SupportedRuntimeIdentifiers.Trim());

I think this would be a more strategic solution (with a comment), as there's no guarantee someone won't change the list to multiline again.

mitchdenny · 2023-05-05T02:54:24Z

@wtgodbe is there a way with the Helix attributes that you want a test to opt into a queue rather than opt out? Seems like in the long term opting out of queues is inefficient for some test cases (although opting out is good for test cases where we know we can achieve broad platform coverage).

tmds · 2023-05-05T07:28:38Z

I think this would be a more strategic solution (with a comment), as there's no guarantee someone won't change the list to multiline again.

The code that reads this is https://github.com/dotnet/sdk/blob/main/src/Tasks/Microsoft.NET.Build.Tasks/ProcessFrameworkReferences.cs. It splits by ; and doesn't ignore whitespace.

I've added a comment that whitespace is not supported in this property.

…ts.in.

tmds · 2023-05-05T10:41:12Z

CI is green.

Also on a Linux system, the aot tests are now passing.

eerhardt

Thanks for finding and fixing this, @tmds!

wtgodbe · 2023-05-05T19:36:55Z

is there a way with the Helix attributes that you want a test to opt into a queue rather than opt out? Seems like in the long term opting out of queues is inefficient for some test cases (although opting out is good for test cases where we know we can achieve broad platform coverage).

Not that I know of - so far there haven't been that many tests that this has been a problem for. We could consider adding that, can you open an issue for it?

ghost added area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates community-contribution Indicates that the PR has been added by a community member labels May 2, 2023

captainsafia removed the community-contribution Indicates that the PR has been added by a community member label May 2, 2023

Tratcher requested review from JamesNK and wtgodbe May 2, 2023 21:27

RussKie reviewed May 4, 2023

View reviewed changes

tmds force-pushed the skip_aot_tests branch from e1315ca to 1f0f23d Compare May 4, 2023 14:29

lewing mentioned this pull request May 4, 2023

[main] Update dependencies from dotnet/runtime dotnet/efcore #48014

Merged

tmds requested a review from a team as a code owner May 5, 2023 07:48

Fix SupportedRuntimeIdentifiers substitution in Directory.Build.targe…

fab2c88

…ts.in.

tmds force-pushed the skip_aot_tests branch from 92196ce to fab2c88 Compare May 5, 2023 09:00

tmds changed the title ~~Skip AOT template tests on Linux, where they are known to fail.~~ Fix SupportedRuntimeIdentifiers substitution in Directory.Build.targets.in. May 5, 2023

eerhardt approved these changes May 5, 2023

View reviewed changes

eerhardt merged commit 801e24f into dotnet:main May 5, 2023

ghost added this to the 8.0-preview5 milestone May 5, 2023

eerhardt mentioned this pull request May 5, 2023

Skip more OS's for AOT tests #48090

Merged

Fix SupportedRuntimeIdentifiers substitution in Directory.Build.targets.in. #48022

Fix SupportedRuntimeIdentifiers substitution in Directory.Build.targets.in. #48022

Uh oh!

Conversation

tmds commented May 2, 2023

Uh oh!

ghost commented May 2, 2023

Uh oh!

eerhardt commented May 2, 2023

Uh oh!

tmds commented May 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eerhardt commented May 2, 2023

Uh oh!

wtgodbe commented May 2, 2023

Uh oh!

Tratcher commented May 2, 2023

Uh oh!

Tratcher commented May 2, 2023

Uh oh!

eerhardt commented May 2, 2023

Uh oh!

Tratcher commented May 2, 2023

Uh oh!

eerhardt commented May 2, 2023

Uh oh!

wtgodbe commented May 2, 2023

Uh oh!

tmds commented May 3, 2023

Uh oh!

wtgodbe commented May 3, 2023

Uh oh!

mitchdenny commented May 4, 2023

Uh oh!

RussKie May 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmds commented May 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RussKie commented May 5, 2023

Uh oh!

mitchdenny commented May 5, 2023

Uh oh!

tmds commented May 5, 2023

Uh oh!

tmds commented May 5, 2023

Uh oh!

eerhardt left a comment

Choose a reason for hiding this comment

Uh oh!

wtgodbe commented May 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

tmds commented May 2, 2023 •

edited

Loading

RussKie May 4, 2023 •

edited

Loading

tmds commented May 4, 2023 •

edited

Loading