Fix warning "Atleast for one assembly start time was not obtained due to tag not present or parsing issue"

[wtgodbe]

Part of #22240

Happens while publishing quarantined test results. Whole warning text:

##[warning]Atleast for one assembly start time was not obtained due to tag not present or parsing issue, total run duration will now be summation of time taken by each assembly
##[warning]Atleast for one assembly time tag is not present, total run duration will now be summation of time from all test runs

[dougbu]

👀

[HaoK]

Meta question, why do we need to run quarantined tests as part of our ci builds at all? Maybe we could just move this job to the helix quarantined pipeline (and make that the dedicated quarantine run for everything not just helix) and worry about warnings there later (if ever)?

[HaoK]

This also has the nice sideffect of getting rid of quarantine test results in our ci runs which is just noise when we see those failures

[dougbu]

If the quarantined test pipeline instead ran every test (quarantined or not) without splitting things in two that might be 🆗 However, I'd be concerned about the additional resources tied up duplicating work done in the rolling builds.

[dougbu]

Simple options we talked about:

• Find a way not to submit Helix jobs that would be empty. This has the nice side effect of reducing our load on Helix queues.
• Add an unconditionally-successful quarantined test to all test assemblies.

In addition, could change our Helix test runner to delete empty test result files -- before the Python stuff tries to submit those results to AzDO.

[HaoK]

I thought this issue is just talking about build warnings/logs. Right now we are running things twice, once in the windows test ci.yml, and again in the helix pipeline

[HaoK]

So we could either just stop running the windows test job quarantine step entirely, or run them both on the helix pipeline if we really want, that's not going to be any worse than what we do today in resources no?

[HaoK]

To be precise, we are doing slightly different things. Helix quarantine pipeline does a helix run of all quarantined tests. The windows test job does a ci build with RunQuarantinedTests = true, its probably a very similar set, but there might be some slighty differences in test results... But given that we only look at flaky test data for helix runs... What's the point of running the windows test job quarantined tests at all?

[HaoK]

Also, this error doesn't exist on helix test jobs, because we don't use the publish test results task, the helix runner scripts directly upload the test results to azdo, so the helix quarantined jobs don't have this warning issue

[dougbu]

We have two categories of quarantined tests: Those that run on Helix and those that don't. If the aspnetcore-quarantined-tests pipeline redoes anything that also runs in aspnetcore-ci or aspnetcore-ci-official, I'm fine reducing what aspnetcore-ci and aspnetcore-ci-official are doing and focusing on the reliable tests there.

I would extend this to aspnetcore-helix-matrix too i.e. don't run quarantined tests anywhere but in the aspnetcore-quarantined-tests pipeline.

But I don't think it's a good idea to transfer "600+" warnings from one pipeline to another and call that an actual improvement.

[HaoK]

Well my main point is that I don't think the non helix quarantined runs are even being looked at, so the win win is to just stop running the non helix quarantined tests completely, the warnings go away, and its not like we are actually looking at those results anyways.

[dougbu]

I disagree with no longer running tests that we have chosen to quarantine and not skip. Someone given an issue to unquarantine on of those tests can always check the AzDO test analytics instead of using the (broken) What's Broken application. This is something we'll need to do anyhow 'til What's Broken isn't broken.

[HaoK]

I still don't think we are quite talking about the same thing. Most quarantined tests are run twice right now, once in the windows test job, and again in the helix-quarantine-pipeline. I would guess azdo test analytics should work on both of those pipelines. So the only tests that would no longer be part of the quarantined runs would be in the handful of test assemblies that aren't running on helix. Those wouldn't have shown up on what's broken analysis either anyways. The only point I'm trying to make, is these warnings are generated by mostly redundant work