Skipped: Template and gRPC tests on ARM

[jkotalik]

There are a bunch of failures that are always happening in the aspnetcore-helix-matrix, mostly involving template tests. These are only happening on the Arm64v8 and Debian.9.Arm64.Open queues.
https://dev.azure.com/dnceng/public/_build/results?buildId=896343&view=ms.vss-test-web.build-test-results-tab&runId=28644850&paneView=debug&resultId=122287

First type of failure:

Project new webapi  --auth IndividualB2C failed to publish. Exit code 1.
D:\h\w\B5BC09B0\w\A83F0910\e\sdk15725\arm64\dotnet.exe publish --no-restore -c Release /bl \nStdErr: \nStdOut: Microsoft (R) Build Engine version 16.9.0-preview-20513-07+c10f34c4e for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

D:\h\w\B5BC09B0\w\A83F0910\e\sdk15725\arm64\sdk\6.0.100-alpha.1.20523.3\MSBuild.dll -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,D:\h\w\B5BC09B0\w\A83F0910\e\sdk15725\arm64\sdk\6.0.100-alpha.1.20523.3\dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,D:\h\w\B5BC09B0\w\A83F0910\e\sdk15725\arm64\sdk\6.0.100-alpha.1.20523.3\dotnet.dll -maxcpucount -property:Configuration=Release -target:Publish -verbosity:m /bl .\AspNet.u2mps5ugpjv.csproj
You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.Authentication.JwtBearer with version (>= 6.0.0-ci)
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 972 version(s) in dotnet5 [ Nearest version: 5.0.0-rtm.20512.8 ]
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 247 version(s) in dotnet6 [ Nearest version: 6.0.0-alpha.1.20570.9 ]
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 66 version(s) in nuget.org [ Nearest version: 5.0.0 ]
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 1 version(s) in dotnet-tools [ Nearest version: 5.0.0-rc.2.20513.11 ]
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 0 version(s) in dotnet-eng
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 0 version(s) in dotnet31-transport
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 0 version(s) in dotnet5-transport
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 0 version(s) in dotnet6-transport
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.Authentication.OpenIdConnect with version (>= 6.0.0-ci)
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 972 version(s) in dotnet5 [ Nearest version: 5.0.0-rtm.20512.8 ]
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 247 version(s) in dotnet6 [ Nearest version: 6.0.0-alpha.1.20570.9 ]
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 66 version(s) in nuget.org [ Nearest version: 5.0.0 ]
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 1 version(s) in dotnet-tools [ Nearest version: 5.0.0-rc.2.20513.11 ]
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 0 version(s) in dotnet-eng
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 0 version(s) in dotnet31-transport
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 0 version(s) in dotnet5-transport
D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.u2mps5ugpjv\AspNet.u2mps5ugpjv.csproj : error NU1102:   - Found 0 version(s) in dotnet6-transport

Expected: True
Actual:   False

gRPC Template failure:

Project new grpc  failed to publish. Exit code 1.
D:\h\w\B5BC09B0\w\A83F0910\e\sdk15725\arm64\dotnet.exe publish --no-restore -c Release /bl \nStdErr: \nStdOut: Microsoft (R) Build Engine version 16.9.0-preview-20513-07+c10f34c4e for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

D:\h\w\B5BC09B0\w\A83F0910\e\sdk15725\arm64\sdk\6.0.100-alpha.1.20523.3\MSBuild.dll -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,D:\h\w\B5BC09B0\w\A83F0910\e\sdk15725\arm64\sdk\6.0.100-alpha.1.20523.3\dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,D:\h\w\B5BC09B0\w\A83F0910\e\sdk15725\arm64\sdk\6.0.100-alpha.1.20523.3\dotnet.dll -maxcpucount -property:Configuration=Release -target:Publish -verbosity:m /bl .\AspNet.retxo3gplgd.csproj
You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
D:\h\w\B5BC09B0\w\A83F0910\e\nugetRestore\grpc.tools\2.32.0\build_protobuf\Google.Protobuf.Tools.targets(84,5): error : Google.Protobuf.Tools cannot determine host OS and CPU. [D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.retxo3gplgd\AspNet.retxo3gplgd.csproj]
D:\h\w\B5BC09B0\w\A83F0910\e\nugetRestore\grpc.tools\2.32.0\build_protobuf\Google.Protobuf.Tools.targets(84,5): error : Use environment variables PROTOBUF_TOOLS_OS={linux|macosx|windows} and PROTOBUF_TOOLS_CPU={x86|x64} to try the closest match to your system. [D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.retxo3gplgd\AspNet.retxo3gplgd.csproj]
D:\h\w\B5BC09B0\w\A83F0910\e\nugetRestore\grpc.tools\2.32.0\build_protobuf\Google.Protobuf.Tools.targets(84,5): error : You may also set PROTOBUF_PROTOC to specify full path to the host-provided compiler (v3.5+ is required). [D:\h\w\B5BC09B0\w\A83F0910\e\Templates\BaseFolder\AspNet.retxo3gplgd\AspNet.retxo3gplgd.csproj]

Expected: True
Actual:   False

Finally a few tests fail to listen.

System.InvalidOperationException : Couldn't find listening url:
   at Templates.Test.Helpers.AspNetProcess.GetListeningMessage() in /_/src/ProjectTemplates/Shared/AspNetProcess.cs:line 241
   at Templates.Test.Helpers.AspNetProcess.ResolveListeningUrl(ITestOutputHelper output) in /_/src/ProjectTemplates/Shared/AspNetProcess.cs:line 197
   at Templates.Test.Helpers.AspNetProcess..ctor(ITestOutputHelper output, String workingDirectory, String dllPath, IDictionary`2 environmentVariables, Boolean published, Boolean hasListeningUri, Boolean usePublishedAppHost, ILogger logger) in /_/src/ProjectTemplates/Shared/AspNetProcess.cs:line 101
   at Templates.Test.Helpers.Project.StartBuiltProjectAsync(Boolean hasListeningUri, ILogger logger) in /_/src/ProjectTemplates/Shared/Project.cs:line 179
   at Templates.Test.MvcTemplateTest.MvcTemplateCore(String languageOverride) in /_/src/ProjectTemplates/test/MvcTemplateTest.cs:line 99
   at Templates.Test.MvcTemplateTest.MvcTemplate_NoAuthCSharp() in /_/src/ProjectTemplates/test/MvcTemplateTest.cs:line 44
--- End of stack trace from previous location ---

Thanks for contacting us.
We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[Tratcher]

Example:

EmptyWebTemplateCSharp
https://dev.azure.com/dnceng/public/_build/results?buildId=902748&view=ms.vss-test-web.build-test-results-tab&runId=28825124&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=121847

[123.588s] [Templates.Test.EmptyWebTemplateTest] [Information] [ERROR] It was not possible to find any compatible framework version
[123.590s] [Templates.Test.EmptyWebTemplateTest] [Information] [ERROR] The framework 'Microsoft.AspNetCore.App', version '6.0.0-ci' was not found.
[123.591s] [Templates.Test.EmptyWebTemplateTest] [Information] [ERROR]   - The following frameworks were found:
[123.593s] [Templates.Test.EmptyWebTemplateTest] [Information] [ERROR]       6.0.0-alpha.1.20513.8 at [/root/helix/work/workitem/.dotnet17030/shared/Microsoft.AspNetCore.App]

Also, this test was disabled a week ago but still seems to be running.

[wtgodbe]

#38427 moves the Debian Arm queue from Debian9 to Debian11, might be worth trying to un-skip the tests there now.

[dougbu]

I'm updating our Arm64 queue for Windows to Windows 11 machines. Initial testing should demonstrate whether the problem is fixed w/ the newer boxes.

[dougbu]

Suspect this issue can be closed now. #44868 merged fine w/o updating the ARM queues mentioned in [SkipOnHelix] attributes. (Yes, we could cleanup [SkipOnHelix] attributes in general but the old queue mentions aren't hurting us.)

I will however be making similar changes to #44868 in release/6.0 and release/7.0. Will see if our luck holds there too.

[dougbu]

I take that ^^ back. Template and gRPC tests did not fair at all well in my release/6.0 version of #44868. Not sure why things are worse in that branch. It is however clear parts of this issue (and some new issues) remain when testing release/6.0 on Win/Arm.