How to run the test until it fails?

[adamsitnik]

When trying to fix a flaky test, I usually need to run it for a while before I am able to reproduce the problem.

Sample command:

.\dotnet.cmd build /t:Test .\src\libraries\System.Diagnostics.Process\tests\System.Diagnostics.Process.Tests.csproj /p:Configuration=Debug /p:TargetArchitecture=x86 /p:XunitMethodName=System.Diagnostics.Tests.ProcessModuleTests.LongModuleFileNamesAreSupported

@ViktorHofer @safern Do we have any built-in way of requesting given test to be run until it fails? If not, would a PR that adds such possibility to dotnet.cmd|sh script be welcomed?

Usually I hack dotnet.cmd and do the following:

+:Loop
call "%dotnetPath%\dotnet.exe" %*
+if %ERRORLEVEL% equ 0 goto :Loop

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

When trying to fix a flaky test, I usually need to run it for a while before I am able to reproduce the problem.

Sample command:

.\dotnet.cmd build /t:Test .\src\libraries\System.Diagnostics.Process\tests\System.Diagnostics.Process.Tests.csproj /p:Configuration=Debug /p:TargetArchitecture=x86 /p:XunitMethodName=System.Diagnostics.Tests.ProcessModuleTests.LongModuleFileNamesAreSupported

@ViktorHofer @safern Do we have any built-in way of requesting given test to be run until it fails? If not, would a PR that adds such possibility to dotnet.cmd|sh script be welcomed?

Usually I hack dotnet.cmd and do the following:

+:Loop
call "%dotnetPath%\dotnet.exe" %*
+if %ERRORLEVEL% equ 0 goto :Loop

Author:	adamsitnik
Assignees:	-
Labels:	question, area-Infrastructure-libraries
Milestone:	Future

[ViktorHofer]

@adamsitnik is there a reason why you use dotnet build /t:Test and not dotnet test? Just curious.

@ViktorHofer @safern Do we have any built-in way of requesting given test to be run until it fails? If not, would a PR that adds such possibility to dotnet.cmd|sh script be welcomed?

We don't today. If the proposed solution works cross-plat and is scoped to test execution only and works with both dotnet test and dotnet build /t:Test (which uses the legacy unsupported xunit.console runner) then for sure. What makes this tricky is that dotnet test doesn't run as part of a cmd/sh script, it invokes dotnet.dll directly.

[adamsitnik]

is there a reason why you use dotnet build /t:Test and not dotnet test? Just curious

@ViktorHofer I can't get it working for x86:

 .\dotnet.cmd test .\src\libraries\System.Diagnostics.Process\tests\System.Diagnostics.Process.Tests.csproj /p:Configuration=Debug /p:TargetArchitecture=x86 /p:XunitMethodName=System.Diagnostics.Tests.ProcessModuleTests.LongModuleFileNamesAreSupported

A total of 1 test files matched the specified pattern.
Testhost process exited with error: Unhandled exception. System.BadImageFormatException: Could not load file or assembly 'D:\projects\runtime\artifacts\bin\System.Diagnostics.Process.Tests\net6.0-windows-Debug\testhost.dll'. An attempt was made to load a program with an incorrect format.
File name: 'D:\projects\runtime\artifacts\bin\System.Diagnostics.Process.Tests\net6.0-windows-Debug\testhost.dll'
. Please check the diagnostic logs for more information.

.\dotnet.cmd test .\src\libraries\System.Diagnostics.Process\tests\System.Diagnostics.Process.Tests.csproj /p:Configuration=Debug /p:XunitMethodName=System.Diagnostics.Tests.ProcessModuleTests.LongModuleFileNamesAreSupported

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Failed to launch testhost with error: System.AggregateException: One or more errors occurred. (An error occurred trying to start process 'D:\projects\runtime\artifacts\bin\testhost\net6.0-windows-Debug-x64\dotnet.exe' with working directory 'D:\projects\runtime\artifacts\bin\System.Diagnostics.Process.Tests\net6.0-windows-Debug'. The system cannot find the file specified.)
 ---> System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'D:\projects\runtime\artifacts\bin\testhost\net6.0-windows-Debug-x64\dotnet.exe' with working directory 'D:\projects\runtime\artifacts\bin\System.Diagnostics.Process.Tests\net6.0-windows-Debug'. The system cannot find the file specified.
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.ProcessHelper.LaunchProcess(String processPath, String arguments, String workingDirectory, IDictionary`2 envVariables, Action`2 errorCallback, Action`1 exitCallBack, Action`2 outputCallBack)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DotnetTestHostManager.LaunchHost(TestProcessStartInfo testHostStartInfo, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DotnetTestHostManager.<>c__DisplayClass38_0.<LaunchTestHostAsync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__271_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Hosting.DotnetTestHostManager.LaunchTestHostAsync(TestProcessStartInfo testHostStartInfo, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources, String runSettings)

[EgorBo]

Unrelated: it'd be nice to have an ability to run any (all) tests some fixed number of times - most tests never reach tier1 so we never test optimized codegen (promoted naturally from tier0) on them with PGO/GDV.

[AndyAyersMS]

One trick you can play if you're trying to repro manually is to find the underlying command line and pass the test assembly multiple times, eg

;; run tests once 
dotnet.exe exec --runtimeconfig System.Net.Security.Tests.runtimeconfig.json --depsfile System.Net.Security.Tests.deps.json xunit.console.dll System.Net.Security.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing

;; run tests twice
dotnet.exe exec --runtimeconfig System.Net.Security.Tests.runtimeconfig.json --depsfile System.Net.Security.Tests.deps.json xunit.console.dll  System.Net.Security.Tests.dll  System.Net.Security.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing

I went up to 32x in some of my recent repro attempts (30 being the magic threshold for Tiered Comp).

[ViktorHofer]

We probably want to make this work. It would be good to tackle this when we have time to clean-up our test runner scripts (use the Arcade provided templating package, add dotnet test to it, merge back the other runner scripts to use the same infrastructure underneath).