Skip to content

Improve flaky tests, move tests blockade. #60930

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Mar 19, 2025

Conversation

ilonatommy
Copy link
Member

Description

  • In EnhancedNavigationScrollBehavesSameAsBrowserOnNavigation tests we mostly care about testing enhanced navigation, not browser navigation. The failing tests are with browser navigation, so it's fine to block only them and leave enhanced-nav based to keep running. I could not find a fix for the failing ones. Failures with System.Exception : Expected to use browser navigation because 'suppress-enhanced-navigation' is set to true but the element from previous path is not stale message might depend on the timeout we use in AssertEnhancedNavigation, so it got bumped.

During the CI investigation, a few other frequent issues were found that might be fixed by changing the assert to a waiting-assert:

[xUnit.net 00:07:02.98]     Microsoft.AspNetCore.Components.E2ETest.ServerExecutionTests.ServerRoutingTest.NavigationToSamePathDoesNotScrollToTheTop [FAIL]
  Failed Microsoft.AspNetCore.Components.E2ETest.ServerExecutionTests.ServerRoutingTest.NavigationToSamePathDoesNotScrollToTheTop [1 s]
  Error Message:
   Expected to scroll to the fragment.
  Stack Trace:
     at Microsoft.AspNetCore.Components.E2ETest.Tests.RoutingTest.NavigationToSamePathDoesNotScrollToTheTop() in /home/vsts/work/1/s/src/Components/test/E2ETest/Tests/RoutingTest.cs:line 583
   at InvokeStub_RoutingTest.NavigationToSamePathDoesNotScrollToTheTop(Object, Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

[xUnit.net 00:10:42.76]     Microsoft.AspNetCore.Components.E2ETest.ServerExecutionTests.ServerRoutingTest.NavigationManagerNavigateToSameUrlWithParamAndHash_ScrollsToElementOnTheSamePage [FAIL]
  Failed Microsoft.AspNetCore.Components.E2ETest.ServerExecutionTests.ServerRoutingTest.NavigationManagerNavigateToSameUrlWithParamAndHash_ScrollsToElementOnTheSamePage [902 ms]
  Error Message:
   Assert.Equal() Failure: Values differ
Expected: 1860
Actual:   0
  Stack Trace:
     at Microsoft.AspNetCore.Components.E2ETest.Tests.RoutingTest.NavigationManagerNavigateToSameUrlWithParamAndHash_ScrollsToElementOnTheSamePage() in /home/vsts/work/1/s/src/Components/test/E2ETest/Tests/RoutingTest.cs:line 1831
   at InvokeStub_RoutingTest.NavigationManagerNavigateToSameUrlWithParamAndHash_ScrollsToElementOnTheSamePage(Object, Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Contributes to #60875

@ilonatommy ilonatommy added test-failure area-blazor Includes: Blazor, Razor Components labels Mar 14, 2025
@ilonatommy ilonatommy added this to the .NET 10 Planning milestone Mar 14, 2025
@ilonatommy ilonatommy self-assigned this Mar 14, 2025
@ilonatommy ilonatommy requested a review from a team as a code owner March 14, 2025 11:30
@ilonatommy ilonatommy enabled auto-merge (squash) March 14, 2025 17:29
@ilonatommy
Copy link
Member Author

ilonatommy commented Mar 18, 2025

On each rerun different, not connected to the change tests fail azdo

attempt 1:
error : (NETCORE_ENGINEERING_TELEMETRY=Test) Test Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.Http2.HandshakeTests.TlsAlpnHandshakeSelectsHttp2From1and2 has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/8bc7329a-8dee-4684-b8cc-9a4c227705ef/workitems/Sockets.FunctionalTests--net10.0/console
error : (NETCORE_ENGINEERING_TELEMETRY=Test) Test Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.Http2.ShutdownTests.GracefulTurnsAbortiveIfRequestsDoNotFinish has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/8bc7329a-8dee-4684-b8cc-9a4c227705ef/workitems/Sockets.FunctionalTests--net10.0/console
error : (NETCORE_ENGINEERING_TELEMETRY=Test) Test Microsoft.AspNetCore.Server.Kestrel.Core.Tests.MessageBodyTests.ReadExitsGivenIncompleteChunkedExtension has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/8bc7329a-8dee-4684-b8cc-9a4c227705ef/workitems/Microsoft.AspNetCore.Server.Kestrel.Core.Tests--net10.0/console

attempt 2:
error : (NETCORE_ENGINEERING_TELEMETRY=Test) Test Microsoft.AspNetCore.Server.Kestrel.Transport.NamedPipes.Tests.NamedPipeConnectionListenerTests.AcceptAsync_ParallelConnections_ClientConnectionsSuccessfullyAccepted has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/45631d55-9f15-41e1-ba9c-0fc01bf2ecc2/workitems/Microsoft.AspNetCore.Server.Kestrel.Transport.NamedPipes.Tests--net10.0/console
error : (NETCORE_ENGINEERING_TELEMETRY=Test) Test Microsoft.AspNetCore.Server.Kestrel.Transport.NamedPipes.Tests.WebHostTests.ListenNamedPipeEndpoint_HelloWorld_ClientSuccess has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/45631d55-9f15-41e1-ba9c-0fc01bf2ecc2/workitems/Microsoft.AspNetCore.Server.Kestrel.Transport.NamedPipes.Tests--net10.0/console
error : (NETCORE_ENGINEERING_TELEMETRY=Test) Test Microsoft.AspNetCore.SignalR.Tests.EndToEndTests.ConnectionCanSendAndReceiveMessages has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/45631d55-9f15-41e1-ba9c-0fc01bf2ecc2/workitems/Microsoft.AspNetCore.SignalR.Tests--net10.0/console
error : (NETCORE_ENGINEERING_TELEMETRY=Test) Test Microsoft.AspNetCore.Server.Kestrel.Core.Tests.ConcurrentPipeWriterTests.QueuesIfFlushIsNotAwaited has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/45631d55-9f15-41e1-ba9c-0fc01bf2ecc2/workitems/Microsoft.AspNetCore.Server.Kestrel.Core.Tests--net10.0/console
error : (NETCORE_ENGINEERING_TELEMETRY=Test) Test Microsoft.AspNetCore.Server.Kestrel.Core.Tests.HeartbeatTests.HeartbeatTakingLongerThanIntervalIsLoggedAsWarning has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/45631d55-9f15-41e1-ba9c-0fc01bf2ecc2/workitems/Microsoft.AspNetCore.Server.Kestrel.Core.Tests--net10.0/console
error : (NETCORE_ENGINEERING_TELEMETRY=Test) Test Microsoft.AspNetCore.Server.Kestrel.Core.Tests.HttpConnectionTests.WriteDataRateTimeoutAbortsConnection has failed. Check the Test tab or this console log: https://helix.dot.net/api/2019-06-17/jobs/45631d55-9f15-41e1-ba9c-0fc01bf2ecc2/workitems/Microsoft.AspNetCore.Server.Kestrel.Core.Tests--net10.0/console

I will mark them all as quarantined, otherwise there's no way to merge it.

@BrennanConroy
Copy link
Member

Don't you think it's a bit suspicious that all of these tests timed out when most or all of them do not have any recent history of flakiness? That smells of an infrastructure issue and not bad tests.

@ilonatommy
Copy link
Member Author

@BrennanConroy, this is possible but then, how do I fix the infra or how do you unblock the CI in such situations so that we can be green?

@ilonatommy
Copy link
Member Author

ilonatommy commented Mar 19, 2025

@BrennanConroy, I changed the issues to be infra ones but BuildAnalysis is red anyway, even though there's 0 non-known test errors.

edit:
What's more, this repo has a bad design of merge mechanism, it is checking all the CI checks, instead of checking the Build Analysis only that is in turn, checking all the checks conditionally (taking into consideration known issues and escape mechanism).

@ilonatommy
Copy link
Member Author

/ba-g BuildAnalysis malfunction, all the errors are already reported

@ilonatommy ilonatommy disabled auto-merge March 19, 2025 10:33
@ilonatommy ilonatommy merged commit a09c099 into dotnet:main Mar 19, 2025
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-blazor Includes: Blazor, Razor Components test-failure
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants