Enable AOT template tests #47247

JamesNK · 2023-03-16T10:34:23Z

Should work now that the api and grpc templates can be published without warnings.

mitchdenny · 2023-03-17T13:47:56Z

/azp run

azure-pipelines · 2023-03-17T13:48:13Z

Azure Pipelines successfully started running 3 pipeline(s).

eerhardt · 2023-03-17T17:52:14Z

Windows is failing with:

error : Platform linker not found. To fix this problem, download and install Visual Studio 2022 from http://visualstudio.com. Make sure to install the Desktop Development for C++ workload. For ARM64 development also install C++ ARM64 build tools. [C:hwA4770944wA7930936eTemplatesBaseFolderAspNet.oki3kcjzakbsAspNet.oki3kcjzakbs.csproj]

We would either need C++ on Helix, or need to run these tests on the build machines.

Linux and OSX are failing with:

NETSDK1082: There was no runtime pack for Microsoft.AspNetCore.App available for the specified RuntimeIdentifier 'linux-x64'.

NETSDK1082: There was no runtime pack for Microsoft.AspNetCore.App available for the specified RuntimeIdentifier 'osx-x64'.

We need to make sure the local runtime packs are being built, and are being used in the tests.

JamesNK · 2023-03-21T01:56:41Z

I don't have the time or build expertise to look at this. I think it should be done to avoid future regressions, but someone else will have to pick it up.

mitchdenny · 2023-03-21T02:39:33Z

I did ask about how we get the Helix images updated with these dependencies in the First Responders ... which apparently was the wrong place ;) I don't know how we update these images either.

wtgodbe · 2023-03-21T16:38:12Z

For the linux/osx tests that fail without the rid-specific SharedFx, that's because we build everything for Helix on windows, then send the bits off to various Helix queues (win, mac, linux), along with the dependencies they need (usually just .dll's). I guess these tests are doing something different and need the live-built SharedFx. A similar problem is tracked by #46620.

My inclination is that, if these tests depend on a SharedFx built on non-windows, then they're not suitable for helix and should be run on the build machines. All you need to do to achieve that is add a [SkipOnHelix("Link to a github issue")]

eerhardt · 2023-03-21T16:50:22Z

There appear to be 2 issues:

On Windows, we need to use a different Helix image.

aspnetcore/eng/targets/Helix.Common.props

Line 53 in f136052

<HelixAvailableTargetQueue Include="Windows.Amd64.Server2022.Open" Platform="Windows" />

We need to be using windows.amd64.vs2022.pre.open or something with "VS" in it.
On Non-Windows, we are not getting the "targeting pack" that is being built for the non-Windows RID as @wtgodbe says above.

My inclination is that, if these tests depend on a SharedFx built on non-windows

These tests depend on the targeting pack. I'm not sure if your statement cares about the difference between that and the installed "SharedFx" or not.

and should be run on the build machines. All you need to do to achieve that is add a [SkipOnHelix("Link to a github issue")]

How does that make the tests run on the build machines? We need the tests to be run somewhere.

eerhardt · 2023-03-21T16:52:13Z

One option here could be to only run the tests on Windows (on Helix) and rely on dotnet/runtime to work correctly on non-Windows, since ASP.NET doesn't have RID-specific code.

That would at least give us some test coverage, which is better than what we have today.

wtgodbe · 2023-03-21T17:08:25Z

How does that make the tests run on the build machines? We need the tests to be run somewhere.

SkipOnHelix is poorly named, it really means "Run on a build agent instead of on Helix".

On Windows, we need to use a different Helix image.

The one you linked is one of the ones we only use in the helix-matrix job, which runs twice per day and mostly only gets looked at by build-ops. The one that runs in CI jobs is here:

aspnetcore/eng/targets/Helix.Common.props

Line 32 in f136052

    
           <HelixAvailableTargetQueue Include="Windows.11.Amd64.Client.Open" Platform="Windows" />

If there's an available Win11 image that has the required VS components, we could switch to that, then add a [SkipOnHelix("{Issue Link}", Queues = "All.OSX;All.Ubuntu" + {Names of win queues that need to be skipped})], or something along those lines.

mitchdenny · 2023-03-29T10:38:32Z

@wtgodbe I'm trying to understand this better. If I put [SkipOnHelix] on a test case, does that mean it will run on an build agent. I've skipped some helix queues for the AOT tests in this PR and I can't see evidence that the tests ran at all.

eerhardt · 2023-03-29T16:01:59Z

I've skipped some helix queues for the AOT tests in this PR and I can't see evidence that the tests ran at all.

An easy test is to explicitly make the test fail and push it. Then see if the test fails CI. If it doesn't, it must not be running.

mitchdenny · 2023-03-30T03:00:32Z

Yep the tests aren't being run.

DamianEdwards · 2023-03-30T13:59:53Z

I thought we didn't run the project templates tests on non-Helix builds at all?

wtgodbe · 2023-03-30T17:18:46Z

@DamianEdwards what makes you say that?

In any case, I think there might be a bug in the SkipOnHelixAttribute. For example, BlazorServerTemplateWorks_IndividualAuth. This one only sets All.OSX as its queues to skip, and in CI it runs on Ubuntu/Windows in Helix, and Windows on a build agent. That seems very odd. As an experiment, could you try adding Queues = "All.Ubuntu" to the SkipOnHelix` attribute?

@halter73 or @BrennanConroy maybe you remember some of the original intention behind SkipOnHelixAttribute? My understanding was that tests that were skipped on helix would instead run on build agents, but it doesn't look like that's the behavior we're getting.

BrennanConroy · 2023-03-30T17:26:04Z

My understanding was that tests that were skipped on helix would instead run on build agents, but it doesn't look like that's the behavior we're getting.

How would it do that? SkipOnHelixAttribute involves a runtime check. I think the way a test runs on Helix or not is determined by the build checking BuildHelixPayload. Which you can see is set to false for the blazor test
https://github.com/dotnet/aspnetcore/blob/83728cac8c2feb5b81f2402c3d83c32f9239c22c/src/ProjectTemplates/test/Templates.Blazor.Tests/Templates.Blazor.Tests.csproj#LL17C31-L17C49

Also, remember, Blazor is a crazy area and has the same named tests in different projects...
So one version of BlazorServerTemplateWorks_IndividualAuth runs on build agents, the other runs on Helix

DamianEdwards · 2023-03-30T17:34:12Z

@wtgodbe probably ignorance 😄

When doing the big template tests overhaul last year (or was it the year before?) it was always the Helix matrix run that was an issue, and I thought I recalled it was because the vast majority of template tests/variations didn't run on the regular CI builds, only on Helix matrix. But I'm likely just misremembering.

wtgodbe · 2023-03-30T17:36:43Z

Huh, I had thought SkipOnHelix was our way of getting an individual test case (rather than the whole test project) to run on a build agent, but it seems I may have been misinformed. Do we not have a way to do that today?

BrennanConroy · 2023-03-30T17:38:04Z

way of getting an individual test case (rather than the whole test project) to run on a build agent

I've never heard of a feature like this, and it sounds like a pain to implement 😄

eerhardt · 2023-03-30T18:26:56Z

What's the recommended approach to getting these tests to run in PR validation?

I would be happy with getting just 1 leg running, for example Windows. Can we have the template test run in a Helix machine with Visual Studio installed? (we will need Visual C++ too)

wtgodbe · 2023-03-30T18:39:01Z

You could change

aspnetcore/eng/targets/Helix.Common.props

Line 53 in f136052

    
           <HelixAvailableTargetQueue Include="Windows.Amd64.Server2022.Open" Platform="Windows" />

to be windows.amd64.vs2022.pre.open, then add Queues = "Windows.11.Amd64.Client.Open;" + All.OSX + All.Ubuntu to the SkipOnHelix attributes. This would get the Windows jobs running in the Helix Matrix pipeline twice daily, but not during CI (I think we can't switch to the vs2022 queue in CI jobs because of capacity, maybe someone from @dotnet/dnceng can confirm)

dougbu · 2023-03-30T19:03:59Z

I think we can't switch to the vs2022 queue in CI jobs because of capacity

The Helix queues auto-balance relatively quickly. As long as it's really a switch, capacity shouldn't be a big deal after the initial ramp up.

mitchdenny · 2023-03-31T00:21:48Z

Yeah we really want this running via CI for obvious reasons. It would be good if you could guide us through the process for using the vs2022.pre queues. I'm assuming its not the same change that @wtgodbe mentioned above.

mitchdenny · 2023-03-31T00:22:20Z

... also is there a config file/doc somewhere that describes what is on each image?

… publish tests on.

JamesNK · 2023-04-21T02:31:05Z

src/Servers/HttpSys/test/FunctionalTests/Http2Tests.cs

@@ -21,6 +21,7 @@

 namespace Microsoft.AspNetCore.Server.HttpSys.FunctionalTests;

+[SkipOnHelix("Unsupported queue", Queues = "Windows.Amd64.VS2022.Pre.Open;")]


Why don't these tests run on this queue? Is HTTP.sys and IIS not on the image?

It would be good to have a better description in the attribute.

Are all the HTTP.sys and IIS functional tests skipped? If so, SkipOnHelix attribute might work on the assembly. Avoids modifying so many files.

Based on the usage attributes the SkipOnHelix attribute doesn't work at the assembly level. The reason I am disabling these tests is that they are causing failtures due to hitting some 40m threshold. I'm not certain if it's that IIS just won't work on that image or if it is something else at this stage.

JamesNK · 2023-04-21T02:36:23Z

src/ProjectTemplates/test/Templates.Tests/ApiTemplateTest.cs

@@ -35,7 +36,8 @@ public async Task ApiTemplateCSharp()
        await ApiTemplateCore(languageOverride: null);
    }

-    [ConditionalFact(Skip = "Unskip when Helix supports native AOT. https://github.com/dotnet/aspnetcore/pull/47247/")]
+    [ConditionalFact]
+    [SkipOnHelix("https://github.com/dotnet/aspnetcore/issues/47478", Queues = "OSX.13.Amd64.Open;Ubuntu.2004.Amd64.Open;Windows.11.Amd64.Client.Open;")]


This set of queues seems duplicated a bunch of times. Maybe have a const in this project and append const here and elsewhere? It makes updating it easier in the future.

mitchdenny · 2023-04-27T00:13:13Z

/azp run

azure-pipelines · 2023-04-27T00:14:53Z

Azure Pipelines successfully started running 3 pipeline(s).

src/ProjectTemplates/test/Templates.Blazor.WebAssembly.Auth.Tests/BlazorWasmTemplateAuthTest.cs

JamesNK added the area-web-frameworks *DEPRECATED* This label is deprecated in favor of the area-mvc and area-minimal labels label Mar 16, 2023

ghost added the area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates label Mar 16, 2023

eerhardt requested review from a team and wtgodbe as code owners March 16, 2023 16:03

wtgodbe approved these changes Mar 16, 2023

View reviewed changes

adityamandaleeka mentioned this pull request Mar 17, 2023

Quarantine MapAction_JsonBodyOrService_HandlesBothJsonAndService #47274

Closed

mitchdenny force-pushed the jamesnk/aot-template-tests-unskip branch from 1a26e9a to c2b4c84 Compare March 29, 2023 02:45

Add Helix queue back in and skip it on tests where it doesn't work.

87c6f76

mitchdenny force-pushed the jamesnk/aot-template-tests-unskip branch from d46fd41 to 87c6f76 Compare April 20, 2023 06:10

mitchdenny requested review from Tratcher, halter73, BrennanConroy and mgravell as code owners April 20, 2023 06:10

mitchdenny added 9 commits April 20, 2023 16:13

Add skipped tests back in that should be fine on Helix.

069bb91

Skip a few more tests that won't run on the agent that we run the AOT…

2b6e995

… publish tests on.

Removed debugging assert.

2205c17

Disable some more functional tests.

ef17e9b

Put fix back in for gRPC tests.

826b052

Disable more IIS tests.

2570397

Add missing namespace.

169541d

Add missing project reference.

cea46ee

Try newer SDK.

70882ef

JamesNK commented Apr 21, 2023

View reviewed changes

mitchdenny modified the milestones: 8.0-preview4, 8.0-preview5 Apr 26, 2023

mitchdenny added 4 commits April 28, 2023 10:19

Merge branch 'main' into jamesnk/aot-template-tests-unskip

741be08

Merge branch 'main' into jamesnk/aot-template-tests-unskip

0d3a4c3

Fix ListCommand.

b725963

Blazor test quarantined.

011444b

JamesNK commented Apr 28, 2023

View reviewed changes

src/ProjectTemplates/test/Templates.Blazor.WebAssembly.Auth.Tests/BlazorWasmTemplateAuthTest.cs Show resolved Hide resolved

mitchdenny merged commit c6ccf23 into main Apr 28, 2023

mitchdenny deleted the jamesnk/aot-template-tests-unskip branch April 28, 2023 08:41

tmds mentioned this pull request May 2, 2023

Fix SupportedRuntimeIdentifiers substitution in Directory.Build.targets.in. #48022

Merged

danmoseley removed the area-web-frameworks *DEPRECATED* This label is deprecated in favor of the area-mvc and area-minimal labels label Jul 7, 2023

Tratcher mentioned this pull request Oct 26, 2023

Failing Test(s): RequestPathBaseTests #51679

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable AOT template tests #47247

Enable AOT template tests #47247

JamesNK commented Mar 16, 2023 •

edited

Loading

mitchdenny commented Mar 17, 2023

azure-pipelines bot commented Mar 17, 2023

eerhardt commented Mar 17, 2023

JamesNK commented Mar 21, 2023

mitchdenny commented Mar 21, 2023

wtgodbe commented Mar 21, 2023

eerhardt commented Mar 21, 2023

eerhardt commented Mar 21, 2023

wtgodbe commented Mar 21, 2023 •

edited

Loading

mitchdenny commented Mar 29, 2023

eerhardt commented Mar 29, 2023

mitchdenny commented Mar 30, 2023

DamianEdwards commented Mar 30, 2023

wtgodbe commented Mar 30, 2023

BrennanConroy commented Mar 30, 2023

DamianEdwards commented Mar 30, 2023

wtgodbe commented Mar 30, 2023

BrennanConroy commented Mar 30, 2023

eerhardt commented Mar 30, 2023

wtgodbe commented Mar 30, 2023 •

edited

Loading

dougbu commented Mar 30, 2023

mitchdenny commented Mar 31, 2023

mitchdenny commented Mar 31, 2023

JamesNK Apr 21, 2023

JamesNK Apr 21, 2023

mitchdenny Apr 21, 2023

JamesNK Apr 21, 2023

mitchdenny commented Apr 27, 2023

azure-pipelines bot commented Apr 27, 2023

		@@ -21,6 +21,7 @@

		namespace Microsoft.AspNetCore.Server.HttpSys.FunctionalTests;

		[SkipOnHelix("Unsupported queue", Queues = "Windows.Amd64.VS2022.Pre.Open;")]

Enable AOT template tests #47247

Enable AOT template tests #47247

Conversation

JamesNK commented Mar 16, 2023 • edited Loading

mitchdenny commented Mar 17, 2023

azure-pipelines bot commented Mar 17, 2023

eerhardt commented Mar 17, 2023

JamesNK commented Mar 21, 2023

mitchdenny commented Mar 21, 2023

wtgodbe commented Mar 21, 2023

eerhardt commented Mar 21, 2023

eerhardt commented Mar 21, 2023

wtgodbe commented Mar 21, 2023 • edited Loading

mitchdenny commented Mar 29, 2023

eerhardt commented Mar 29, 2023

mitchdenny commented Mar 30, 2023

DamianEdwards commented Mar 30, 2023

wtgodbe commented Mar 30, 2023

BrennanConroy commented Mar 30, 2023

DamianEdwards commented Mar 30, 2023

wtgodbe commented Mar 30, 2023

BrennanConroy commented Mar 30, 2023

eerhardt commented Mar 30, 2023

wtgodbe commented Mar 30, 2023 • edited Loading

dougbu commented Mar 30, 2023

mitchdenny commented Mar 31, 2023

mitchdenny commented Mar 31, 2023

JamesNK Apr 21, 2023

Choose a reason for hiding this comment

JamesNK Apr 21, 2023

Choose a reason for hiding this comment

mitchdenny Apr 21, 2023

Choose a reason for hiding this comment

JamesNK Apr 21, 2023

Choose a reason for hiding this comment

mitchdenny commented Apr 27, 2023

azure-pipelines bot commented Apr 27, 2023

JamesNK commented Mar 16, 2023 •

edited

Loading

wtgodbe commented Mar 21, 2023 •

edited

Loading

wtgodbe commented Mar 30, 2023 •

edited

Loading