Test PackageUpdateResource_SourceAndSymbolNuGetOrgPushingAsync is flaky

[zivkan]

The test NuGet.Protocol.Tests.PackageUpdateResourceTests.PackageUpdateResource_SourceAndSymbolNuGetOrgPushingAsync appears to make real network connections to nuget.org and nuget.smbsrc.net. it's just flaky

I don't think our CI should block PRs being merged if an external resource has a service outage, or the CI agent is having flakey network access to the outside world.

[donnie-msft]

Hit this https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=4614095&view=ms.vss-test-web.build-test-results-tab&runId=20308800&resultId=104398&paneView=debug (failure no longer available)

[dominoFire]

Another hit: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=4667107&view=ms.vss-test-web.build-test-results-tab&runId=20821490&resultId=104351&paneView=debug

[kartheekp-ms]

NuGet.Protocol.Tests.PackageUpdateResourceTests.PackageUpdateResource_SourceAndSymbolNuGetOrgPushingAsync test has failed in three 1, 2, 3 official builds recently.

One commonality in all the failures is the test is failing for symbolSource: https://nuget.smbsrc.net/ input.

https://github.com/NuGet/NuGet.Client/blob/d2bb0de35242b04802c1a7856ebb7718f674565c/test/NuGet.Core.Tests/NuGet.Protocol.Tests/Resources/PackageUpdateResourceTests.cs#L226-L231

[donnie-msft]

Took a quick look but not sure what we want to do here. If we Mock, then it no longer verifies how expected responses are handled by Push.
We could skip the test, skip only the "faulty" symbol source, add support for HTTP retries, ...other?

Also when I ran fiddler, this didn't emit any HTTP traffic that I could see.

[zivkan]

I took a quick look myself.

Firstly, I was wrong about it making real network connections. The test uses StaticHttpHandler.CreateSource, which implementes a `HttpClientHandler, which looks up URLs and pre-canned responses in a dictionary which the test itself provides.

Having written all that, I don't currently have any ideas why the test sometimes times out. The test itself specifies 5 second time out, and given that NuGet.Protocol tries requests 3 times by default, that explains the 15 second duration when tests fail.

What I'm confident about is that repeating the test 4 times is pointless.

It would be better for the tests to use a fake domain name ending in .test, to make it clear that it's not making real HTTP requests.

Given all this, I'm going to skip this test because it's too flaky at the moment.