dotnet test: warning: Overwriting results file: duplicate trx file

[kanchm]

Description

When we start up tests from a solution file using dotnet test -l trx, we encounter this warning occasionally:
EXEC : warning : Overwriting results file: D:\...\TestResults\2020.07.17-17.41.02\...04231817_2020-07-17_17_41_25.trx [d:\dbs\sh\acaa\0717_171134\cmd\0\src\dirs.proj]

Problem

The failure is stemming from SetDefaultTrxFilePath computing the next available file name through IsValidIteration but does not reserve it. In case two test sessions are completing at the same time, they end up with the same trx filename, and by the time they write the results, one ends up clobbering the other.

Fix

Expect that the file gets created in IsValidIteration (and suppress the overwrite warning).

[Sanan07]

@kanchm Would it be okay for you if the content of file will be appended instead of overwriting it?

[nohwnd]

@Sanan07 gave this to Medeni a bit ago, but forgot to assign it please sync.

[kanchm]

Would appending mean merge the two XML files? File level append is not an option. Also this would be unsafe. The two independent appends would still step on each other.

An approach is to use Path.GetTempFileName which would reserve the name by creating the file and letting the trx writer overwrite it. You would not warn on the overwrite in this scenario.

Also, related to this is LogFilePrefix which could be improved by adding milliseconds "ffff" as I hit collisions at a second granularity when I tried it.

[Haplois]

Actually I am thinking to append a number to end of it, if there is a collusion. In this case it could be 04231817_2020-07-17_17_41_25-1.trx and 04231817_2020-07-17_17_41_25.trx for example.

[Sanan07]

In this case you should take care of several possibilities of the same file collision, so you should store mapping for each file name.
E.g you already have file logFile.trx, then you have collision, so you create logFile-1.trx, then you again have collision, so in this case you should create logFile-2.trx and etc.
So in my opinion adding milliseconds should solve the issue, or add some randomly generated number as a suffix.

[nohwnd]

I am not sure if we rely on the naming somewhere.

If we do, I would consider checking if the name already exists, and adding 1s to the name, until we find a non-conflicting name:

e.g 04231817_2020-07-17_17_41_25.trx -> 04231817_2020-07-17_17_41_26.trx

if we don't, then adding the milliseconds as suggested above. if we are changing how the file is named, adding the ms is probably better than adding a suffix.

[Haplois]

@Sanan07 of course I'll be checking for the first available number and not just append -1 to the end of it.

@nohwnd as far as I see we don't. I am checking though. I don't like the idea of adding a second (because it will be misleading of actual log start time) or moving to millisecond granularity mainly because it would not fix the problem, just make it harder to occur.

[nohwnd]

@ShreyasRmsft do you depend on the trx name in azdo? Or do you think we are free to change the format? I would guess some users are depending on it in their own pipelines, what do you think?

@Haplois

because it will be misleading of actual log start time

I don't think I would ever notice unless this occurs a lot of times in the row an pushes my timestamp by a full minute.

moving to millisecond granularity mainly because it would not fix the problem, just make it harder to occur.

You can "wait" 1 ms and try checking the filename again, then it won't happen ever, unless there is result file written every ms, which is veeery unlikely.

[ShreyasRmsft]

@nohwnd there are tasks that depend on trx naming conventions when uploading results, my suggestion is to not make any changes to the defaults but use cmdline paramters to modify the naming conventions if needed.

[Haplois]

@ShreyasRmsft thank you for the warning. In that case we might go with @nohwnd's suggestion IMO.