Fix flakiness/timeout in gRPC template tests #19982

JunTaoLuo · 2020-03-19T03:12:18Z

Do not search for port number for cases where we are testing for failure.

I don't know how these tests ever passed on macOS.

JamesNK · 2020-03-19T03:59:07Z

I don't know how these tests ever passed on macOS.

Did the change in #19252 do it?

JunTaoLuo · 2020-03-19T05:25:48Z

Ah that's probably it.

BrennanConroy · 2020-03-19T06:19:14Z

You may have fixed the test failure flakiness, but you haven't fixed the hang.

JunTaoLuo · 2020-03-19T18:07:32Z

Argghh, I'll see what's causing the hang now.

Pilchie · 2020-03-19T18:18:51Z

👀

Do not search for port number for cases where we are testing for failure.

This reverts commit c49f169.

JunTaoLuo · 2020-03-25T06:30:58Z

FYI don't review yet, still testing a few hypotheses.

This reverts commit 00a4267.

dougbu · 2020-03-25T17:35:42Z

Can we please skip these tests while investigating?

BrennanConroy · 2020-03-25T17:37:01Z

Can we please skip these tests while investigating?

?? These tests are skipped in the main runs...

dougbu · 2020-03-25T17:39:35Z

Aren't the tests causing quarantined runs to timeout?

BrennanConroy · 2020-03-25T17:40:29Z

Were causing, hence they were skipped.

JunTaoLuo · 2020-03-25T17:55:57Z

I have fixed the tests already, I'm just trying to pin down a more specific cause for the test hangs. And no, the whole point is to figure out what was happening in these tests so we shouldn't be skipping them.

JunTaoLuo · 2020-03-25T20:20:31Z

Okay with the latest test failure (https://dev.azure.com/dnceng/public/_build/results?buildId=573237&view=ms.vss-test-web.build-test-results-tab&runId=17971944&paneView=debug&resultId=100107), I think I've confirmed that:

The underlying issue is that we were trying to scrape a port in the test case where we expect failure, this can cause a hang, but there seems to be a race since sometimes the test might timeout and fail instead.
While fixing 1. I created a new issue where we were not waiting for process exit in the test cases where we expected failure. This manifested as additional flakiness with the updated runtime but appeared as a hang with the old runtime.

Having fixed both issues, I think we are ready to run these tests under quarantine again.

BrennanConroy · 2020-03-25T21:03:28Z

src/ProjectTemplates/test/GrpcTemplateTest.cs

        [SkipOnHelix("Not supported queues", Queues = "Windows.7.Amd64;Windows.7.Amd64.Open;OSX.1014.Amd64;OSX.1014.Amd64.Open")]
        [QuarantinedTest("https://github.com/dotnet/aspnetcore/issues/19716")]
        public async Task GrpcTemplate()
        {
+            // Setup AssemblyTestLog
+            var assemblyLog = AssemblyTestLog.Create(Assembly.GetExecutingAssembly(), baseDirectory: Project.ArtifactsLogDir);


Why are we doing this here instead of the normal way?

Glad you asked. Normally, we would be using LoggedTest but that requires the use of AspNetTestFramework xunit test framework. This cannot be used in template tests since they already use their own test framework:

aspnetcore/src/ProjectTemplates/test/AssemblyInfo.AssemblyFixtures.cs

Line 8 in 9f1a978

[assembly: TestFramework("Microsoft.AspNetCore.E2ETesting.XunitTestFrameworkWithAssemblyFixture", "ProjectTemplates.Tests")]

and so we have to do it the oldschool way.

I think it's possible to combine the logic of the two xunit frameworks but that's a rather large undertaking that doesn't seem appropriate here.

BrennanConroy

The test log stuff seems odd, but I like the overall change

JunTaoLuo requested a review from BrennanConroy March 19, 2020 03:12

ghost added the area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates label Mar 19, 2020

JunTaoLuo force-pushed the johluo/fix-grpc-template branch from 642ca2c to f89277a Compare March 19, 2020 03:13

JunTaoLuo added area-grpc Includes: GRPC wire-up, templates and removed area-mvc Includes: MVC, Actions and Controllers, Localization, CORS, most templates labels Mar 19, 2020

JunTaoLuo force-pushed the johluo/fix-grpc-template branch from f89277a to 0910388 Compare March 23, 2020 08:28

JunTaoLuo requested review from dougbu and SteveSandersonMS as code owners March 25, 2020 04:39

John Luo added 5 commits March 24, 2020 21:59

Fix flakiness/timeout in gRPC template tests

dd80b39

Do not search for port number for cases where we are testing for failure.

Debug add test logging for hangs

6c130c1

Log to relative dir

b6f7b73

Revert "Log to relative dir"

cdf7de3

This reverts commit c49f169.

Maybe it's exiting too early

051bfc5

JunTaoLuo force-pushed the johluo/fix-grpc-template branch from 4c1bcc2 to 9a2de83 Compare March 25, 2020 05:01

Test theory that the hang was caused by runtime changes

00a4267

JunTaoLuo force-pushed the johluo/fix-grpc-template branch from 9a2de83 to 00a4267 Compare March 25, 2020 05:53

JunTaoLuo requested review from analogrelay and Tratcher as code owners March 25, 2020 05:53

JunTaoLuo removed request for analogrelay, SteveSandersonMS and Tratcher March 25, 2020 06:30

John Luo added 2 commits March 25, 2020 00:16

Revert "Test theory that the hang was caused by runtime changes"

b6c16df

This reverts commit 00a4267.

Test theory that the hang was actually fixed

108eff3

Test again

a8d6b78

Test again

dd93d2e

Add fix back

3bf6852

JunTaoLuo requested review from BrennanConroy and removed request for BrennanConroy and dougbu March 25, 2020 20:21

Copy pasta

b50b337

BrennanConroy reviewed Mar 25, 2020

View reviewed changes

BrennanConroy approved these changes Mar 25, 2020

View reviewed changes

analogrelay added this to the 5.0.0-preview3 milestone Mar 25, 2020

JunTaoLuo merged commit 20f6d65 into master Mar 25, 2020

JunTaoLuo deleted the johluo/fix-grpc-template branch March 25, 2020 23:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix flakiness/timeout in gRPC template tests #19982

Fix flakiness/timeout in gRPC template tests #19982

Uh oh!

JunTaoLuo commented Mar 19, 2020

Uh oh!

JamesNK commented Mar 19, 2020

Uh oh!

JunTaoLuo commented Mar 19, 2020

Uh oh!

BrennanConroy commented Mar 19, 2020

Uh oh!

JunTaoLuo commented Mar 19, 2020

Uh oh!

Pilchie commented Mar 19, 2020

Uh oh!

JunTaoLuo commented Mar 25, 2020

Uh oh!

dougbu commented Mar 25, 2020

Uh oh!

BrennanConroy commented Mar 25, 2020

Uh oh!

dougbu commented Mar 25, 2020

Uh oh!

BrennanConroy commented Mar 25, 2020

Uh oh!

JunTaoLuo commented Mar 25, 2020

Uh oh!

JunTaoLuo commented Mar 25, 2020

Uh oh!

BrennanConroy Mar 25, 2020

Uh oh!

JunTaoLuo Mar 25, 2020

Uh oh!

BrennanConroy left a comment

Uh oh!

Uh oh!

Fix flakiness/timeout in gRPC template tests #19982

Fix flakiness/timeout in gRPC template tests #19982

Uh oh!

Conversation

JunTaoLuo commented Mar 19, 2020

Uh oh!

JamesNK commented Mar 19, 2020

Uh oh!

JunTaoLuo commented Mar 19, 2020

Uh oh!

BrennanConroy commented Mar 19, 2020

Uh oh!

JunTaoLuo commented Mar 19, 2020

Uh oh!

Pilchie commented Mar 19, 2020

Uh oh!

JunTaoLuo commented Mar 25, 2020

Uh oh!

dougbu commented Mar 25, 2020

Uh oh!

BrennanConroy commented Mar 25, 2020

Uh oh!

dougbu commented Mar 25, 2020

Uh oh!

BrennanConroy commented Mar 25, 2020

Uh oh!

JunTaoLuo commented Mar 25, 2020

Uh oh!

JunTaoLuo commented Mar 25, 2020

Uh oh!

BrennanConroy Mar 25, 2020

Choose a reason for hiding this comment

Uh oh!

JunTaoLuo Mar 25, 2020

Choose a reason for hiding this comment

Uh oh!

BrennanConroy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!