HELIX failures investigations #41563

DamianEdwards · 2022-05-06T21:36:00Z

Investigating helix failures.

dougbu · 2022-05-09T16:49:11Z

Suggest adding CollectDump to the --blame option at https://github.com/dotnet/aspnetcore/blob/main/eng/tools/HelixTestRunner/TestRunner.cs#L224

May also be worthwhile adding VSTEST_ DISABLE_FASTER_JSON_SERIALIZATION=1 and DOTNET_CLI_VSTEST_TRACE=1 after `https://github.com/dotnet/aspnetcore/blob/main/eng/tools/HelixTestRunner/TestRunner.cs#L37

src/ProjectTemplates/Shared/Project.cs

src/ProjectTemplates/test/BaselineTest.cs

dougbu · 2022-05-09T17:34:19Z

/fyi @MarcoRossignoli this PR incorporates your suggestions

src/ProjectTemplates/Shared/ProjectFactoryFixture.cs

src/ProjectTemplates/test/BlazorServerTemplateTest.cs

MarcoRossignoli · 2022-05-10T08:11:08Z

cc: @nohwnd

nohwnd · 2022-05-11T17:31:30Z

I had a look on your logs, and there is no indication of what is happening, other than that a template test always runs when crash happens. Testhost seems to abruptly die.

So there are few things you can do:

Try enabling --blame-crash to see if you can capture a crash dump. the datacollector.log will have useful info about why it failed to capture the dump. If you are targetting <net5 tfm you should have Procdump in your PATH. Azure has a Choco task that can install it easily.
If the crash happens on Windows, I would try to get few of the last Application Error logs from Event Log they have useful info.

get-eventlog -LogName application -entrytype error -Source "application error" | Select -First 10 | Format-List

Are you by any chance connecting the processes into a Job (group of processes), so when one process dies the others follow?

src/ProjectTemplates/test/BlazorServerTemplateTest.cs

HaoK · 2022-05-11T21:55:38Z

@nohwnd so what we are trying in this PR is:

var commonTestArgs = $"test {Options.Target} --diag:{diagLog} --logger:xunit --logger:\"console;verbosity=normal\" --blame \"CollectHangDump;CollectDump;TestTimeout=15m\"";

It looks like some new blame variants have been added like the one you suggested, can we drop our blame with CollectHangDump/Dump/TestTimeout and replace with blame-crash and blame-hang-timeout:

var commonTestArgs = $"test {Options.Target} --diag:{diagLog} --logger:xunit --logger:\"console;verbosity=normal\" --blame-crash --blame-hang-timeout \"15m\"";

DamianEdwards · 2022-05-11T22:00:15Z

@HaoK @nohwnd I've updated it here 3af7352

nohwnd · 2022-05-12T08:15:49Z

@HaoK the newer --blame-crash syntax translates to what you already had. I did not see you are already using CollectDump in the code, I probably overlooked it or saw the code before it was added.

In the latest build with abort, the vstest logs look like processes are killed externally, and this log indicates that this is the case:

2022-05-12T00:27:28.8279660Z Non-quarantined tests exceeded configured timeout: 40m.

So I guess you have one more timeout to change? :)

- Ensure project directory name uniqueness - Comment out DotNetNewLock for now

…project template tests

- Deleted unused code - Updated README with details about test timeouts & conditional tests

eng/tools/HelixTestRunner/TestRunner.cs

HaoK

Thanks for grinding through this even after falling into all the pits of doom, things look way better!

BrennanConroy · 2022-05-12T20:18:34Z

src/ProjectTemplates/README.md

@@ -62,6 +62,101 @@ Then, use one of:
 previous step, it is NOT advised that you install templates created on your local machine using just
 `dotnet new -i [nupkgPath]`.

+#### Conditional tests & skipping test platforms


How much of this is specific to template tests? Should it be documented in another location instead?

Yeah @HaoK said the same thing in a comment on a commit before I rebased (so it got lost from here) but I'll do that shuffle once this is in main.

Maybe docs/Helix.md for most of this❔

dougbu

One thing to clean up and a few questions…

src/ProjectTemplates/Shared/ProjectFactoryFixture.cs

src/ProjectTemplates/test/ArgConstants.cs

src/ProjectTemplates/test/BlazorServerTemplateTest.cs

dougbu · 2022-05-12T21:52:07Z

src/ProjectTemplates/test/BlazorServerTemplateTest.cs

+    [ConditionalTheory]
+    [InlineData("Individual", new string[] { ArgConstants.UseLocalDb })]
+    [InlineData("Individual", new string[] { ArgConstants.UseProgramMain, ArgConstants.UseLocalDb })]
+    [OSSkipCondition(OperatingSystems.Linux | OperatingSystems.MacOSX, SkipReason = "No LocalDb on non-Windows")]


Any idea how these tests ever succeeded on Linux or macOS before❔

The pages don't fail unless you try to actually register a user (i.e. hit the database).

src/ProjectTemplates/test/ProjectTemplates.Tests.csproj

dougbu · 2022-05-12T21:56:23Z

src/ProjectTemplates/README.md

@@ -62,6 +62,101 @@ Then, use one of:
 previous step, it is NOT advised that you install templates created on your local machine using just
 `dotnet new -i [nupkgPath]`.

+#### Conditional tests & skipping test platforms


Maybe docs/Helix.md for most of this❔

DamianEdwards added the * NO MERGE * Do not merge this PR as long as this label is present. label May 6, 2022

DamianEdwards requested a review from Pilchie as a code owner May 6, 2022 21:36

ghost added the area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates label May 6, 2022

DamianEdwards requested review from a team, dougbu and wtgodbe as code owners May 9, 2022 16:30

dougbu mentioned this pull request May 9, 2022

Disable test parallelization for project template tests #41587

Closed

dougbu reviewed May 9, 2022

View reviewed changes

src/ProjectTemplates/Shared/Project.cs Outdated Show resolved Hide resolved

src/ProjectTemplates/test/BaselineTest.cs Outdated Show resolved Hide resolved

dougbu requested a review from HaoK May 9, 2022 17:28

HaoK reviewed May 9, 2022

View reviewed changes

src/ProjectTemplates/Shared/ProjectFactoryFixture.cs Outdated Show resolved Hide resolved

HaoK reviewed May 9, 2022

View reviewed changes

src/ProjectTemplates/test/BlazorServerTemplateTest.cs Outdated Show resolved Hide resolved

captainsafia mentioned this pull request May 10, 2022

Run tests on Windows again #41599

Merged

HaoK reviewed May 11, 2022

View reviewed changes

src/ProjectTemplates/test/BlazorServerTemplateTest.cs Outdated Show resolved Hide resolved

DamianEdwards added 10 commits May 12, 2022 09:55

Update template baseline test to use original file name

3018652

Comment out tenmplate baseline test namespace declaration

3c2b8d0

Comment out template warning checks

f038bb5

Trigger matrix helix builds for this branch

ad357c5

Enable more tracing & dump collect in helix test runner

82217ae

Disable test parallelization for project template tests

64e1e78

Update Web API template test to ensure unique project key

70a7049

Remove parallelism from template tests

7082f66

- Ensure project directory name uniqueness - Comment out DotNetNewLock for now

Fix Blazor WASM template tests

3872c14

Fix xunit warning

19d6783

HaoK and others added 9 commits May 12, 2022 09:55

Bump test timeout to 60m to see if that helps

05edb44

Print test timeout message & prepare for clean-up

643c693

Ignore process exit code when printing CTS timeout

0d0dd5b

Add timestamps to helix runner console logs & bump helix timeout for …

039e163

…project template tests

Skip LocalDb template tests on non-Windows platforms

bc1d6a1

Actually skip tests

0614b27

Update test cmd args in TestRunner.cs

94e1d63

Split up long-running theory to avoid test timeouts

e44846b

Remove template test locks & skip template tests on Debian11Arm

e06b8d7

- Deleted unused code - Updated README with details about test timeouts & conditional tests

DamianEdwards force-pushed the damianedwards-template-tests branch from 2f51c1b to e06b8d7 Compare May 12, 2022 16:55

Revert helix matrix config

9bff3cc

BrennanConroy reviewed May 12, 2022

View reviewed changes

eng/tools/HelixTestRunner/TestRunner.cs Outdated Show resolved Hide resolved

HaoK approved these changes May 12, 2022

View reviewed changes

PR feedback

0f5be08

BrennanConroy reviewed May 12, 2022

View reviewed changes

BrennanConroy approved these changes May 12, 2022

View reviewed changes

DamianEdwards enabled auto-merge (squash) May 12, 2022 20:32

dougbu suggested changes May 12, 2022

View reviewed changes

PR feedback

7cc446e

dougbu approved these changes May 12, 2022

View reviewed changes

DamianEdwards merged commit cab4c6e into main May 13, 2022

DamianEdwards deleted the damianedwards-template-tests branch May 13, 2022 00:36

ghost added this to the 7.0-preview5 milestone May 13, 2022

dougbu removed the * NO MERGE * Do not merge this PR as long as this label is present. label May 13, 2022

wtgodbe mentioned this pull request Feb 2, 2023

Update Debian11 Arm64 queue to Debian12 #46419

Merged

wtgodbe mentioned this pull request Jul 19, 2023

PlatformNotSupported exceptions in WindowsARM infrastructure for release/6.0 #49061

Closed

HELIX failures investigations #41563

HELIX failures investigations #41563

Uh oh!

Conversation

DamianEdwards commented May 6, 2022

Uh oh!

dougbu commented May 9, 2022

Uh oh!

Uh oh!

Uh oh!

dougbu commented May 9, 2022

Uh oh!

Uh oh!

Uh oh!

MarcoRossignoli commented May 10, 2022

Uh oh!

nohwnd commented May 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

HaoK commented May 11, 2022

Uh oh!

DamianEdwards commented May 11, 2022

Uh oh!

nohwnd commented May 12, 2022

Uh oh!

Uh oh!

HaoK left a comment

Choose a reason for hiding this comment

Uh oh!

BrennanConroy May 12, 2022

Choose a reason for hiding this comment

Uh oh!

DamianEdwards May 12, 2022

Choose a reason for hiding this comment

Uh oh!

dougbu May 12, 2022

Choose a reason for hiding this comment

Uh oh!

dougbu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dougbu May 12, 2022

Choose a reason for hiding this comment

Uh oh!

DamianEdwards May 12, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dougbu May 12, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nohwnd commented May 11, 2022 •

edited

Loading