Unblock run when cancellation was requested #5267

nohwnd · 2025-03-19T13:12:32Z

Ensure that we can complete the run when non-graceful cancellation is requested (e.b. ctrl+C on commandline), by making sure that the test runner does not wait for all currently running tests to finish.

…ng tests

nohwnd · 2025-03-19T13:13:26Z

src/Adapter/MSTest.TestAdapter/Execution/TestExecutionManager.cs

+                    // They will still run on background but we won't await them.
+                    var runCancelled = new TaskCompletionSource<object>();
+                    _testRunCancellationToken?.Register(() => runCancelled.TrySetCanceled());
+                    await Task.WhenAny(Task.WhenAll(tasks), runCancelled.Task);


Cancellation token _testRunCancellationToken is not a CancellationToken, so we cannot use Task.WhenAll(tasks, ct) here.

nohwnd · 2025-03-19T15:42:34Z

test/IntegrationTests/MSTest.Acceptance.IntegrationTests/AbortionTests.cs

        string fileCreationPath = Path.Combine(testHost.DirectoryName, "fileCreation");
-        File.WriteAllText(fileCreationPath, string.Empty);


This was unblocking the loop below, so as soon as we started the app we cancelled tests before any test ever had a chance to run.

So this test was testing almost nothing.

nohwnd · 2025-03-19T15:43:15Z

test/IntegrationTests/MSTest.Acceptance.IntegrationTests/AbortionTests.cs


-        testHostResult.AssertOutputMatchesRegex("Canceling the test session.*");


Now we abort so fast that this does not have a chance to write to screen.

nohwnd · 2025-03-19T15:45:59Z

src/Adapter/MSTest.TestAdapter/Execution/TestExecutionManager.cs

                await ExecuteTestsWithTestRunnerAsync(nonParallelizableTestSet, frameworkHandle, source, sourceLevelParameters, testRunner, usesAppDomains);
            }
        }
        else
        {
+            // What to do?


Here and few lines above we run the tests when parallelization is not enabled or test is marked with do not parallelize, this blocks the cancellation until the test finishes (and fails the non-parallel tests), and advice here how elegantly solve this? (it won't block if we have parallelism with 1 worker, because that takes the other parallel path.

@nohwnd How does it block cancellation? ExecuteTestsWithTestRunnerAsync calls ThrowIfCancellationRequested

Yes but only before it starts running the test. if we cancel while the test is running we don't have a way to abandon it to background and continue to test run teardown code. Like writing test summary to screen. We still have 1 tests in progress and wait for it to complete.

You can still do the same Task.WhenAny trick right?

yeah 😀 I must be tired

Youssef1313 · 2025-03-20T12:11:43Z

src/Adapter/MSTest.TestAdapter/Execution/TestExecutionManager.cs

+        // This method does a lot of work synchronously and cannot be easily cancelled. Make sure we run asynchronously and
+        // can abandon the work here (by calling Task.WhenAny higher in the call stack) when user cancels the test run.
+        await Task.Yield();


We are already calling ThrowIfCancellationRequested just before we call this method. I'm failing to understand why we need Task.Yeild here

ThrowIfCancellationRequested will throw when cancellation was requested before we see that line of code. It has no impact on code that is already running.

So if we cancel before entering this method, all is good and we will cancel the process and will be able to leave the Main method and teardown the process.

But if we enter this method, it will run synchronously, and we are forced to await its completion. So as long as there is a test running (e.g. test that is hanging) we won't be able to leave that stack and won't be able to complete the Main method. The Task.WhenAny that we called this method from will not be effective because the method is running synchronously.

This Yield makes sure that this method runs asynchronously, so we can avoid awaiting its completion.

Another approach would be to merge how tests are run when parallel is enabled and when parallel is disabled. When parallel is enabled (even if there is single worker allowed = serial execution) we offload the work to a Task in a queue, which is what this does as well.

But I think I am just implementing a partially graceful cancellation here, because we need cooperation from the code that runs the tests (not the test themselves, but the code that runs them), so NUnit would also have to go and make sure they cancel in a way that allows them to finish the process.

Instead I should maybe lift this approach higher, and yield in place where we start the tests in platform. So platform ran re-gain control on ctrl+c, without any cooperation with the test framework.

Youssef1313 · 2025-03-20T12:12:09Z

src/Adapter/MSTest.TestAdapter/Execution/TestExecutionManager.cs

@@ -487,16 +490,18 @@ private async Task ExecuteTestsInSourceAsync(IEnumerable<TestCase> tests, IRunCo
            // Queue the non parallel set
            if (nonParallelizableTestSet != null)
            {
+                _testRunCancellationToken?.ThrowIfCancellationRequested();


nit: As you have the same line in the if/else, consider moving it once before the if

Evangelink · 2025-03-20T09:17:40Z

src/Adapter/MSTest.TestAdapter/Execution/TestExecutionManager.cs

-                    await Task.WhenAll(tasks);
+                    // Ensure we can abandon currently running tasks on cancellation, rather than waiting for them to complete.
+                    // They will still run on background but we won't await them.
+                    await Task.WhenAny(Task.WhenAll(tasks), _testRunCancellationToken.AsTask());


I would make the Task.WhenAny(task, cancellationToken.AsTask()) part as the helper instead. Also, could we - if not already done - get some blessing from runtime? I saw multiple different suggestions for this online.

Youssef1313 · 2025-03-21T10:36:40Z

src/Adapter/MSTest.TestAdapter/Execution/TestExecutionManager.cs

+                _testRunCancellationToken?.ThrowIfCancellationRequested();
+                // Ensure we can abandon currently running tasks on cancellation, rather than waiting for them to complete.
+                // They will still run on background but we won't await them.
+                await Task.WhenAny(ExecuteTestsWithTestRunnerAsync(nonParallelizableTestSet, frameworkHandle, source, sourceLevelParameters, testRunner, usesAppDomains), completeWhenTestRunIsCanceled);


The more I think of it, the more I think it's probably not the right thing. Cancellation will be requested, then we just proceed while the task of ExecuteTestsWithTestRunnerAsync could still be running.

I agree...you don't know what the dangling tasks will do and when...at this point it's better Env.Exit(abort code) the result of the run cannot be trusted and some other crash could happen for race for unexpected app state after this return the behavior will be always undefined.

Or you cooperate or you don't cooperate or user should decide what to do --disable-cooperative-cancellation and Env.Exit(abort code), by default cooperative and wait.

Evangelink · 2025-03-23T08:54:06Z

@nohwnd could you please summarize the use case you are trying to fix here?

I seem to see 2 cases:

We want to exit but we want to allow cleanups (platform, extensions, framework...) in which case current implementation (no modification) is the best and this does match cooperative cancellation. Yes it's possible that you end up with a super long cleanup, it's possible that we have a bug in the platform or some extension that does not play well with cooperative and so needs to be fixed.
We want a fast exit, in which case we should end the process (Env fail fast for example) BUT we cannot guarantee cleanup.

From looking up online, it seems that MOST apps/people expect the CTRL+C to be an abortion and not a cancellation so maybe the only change we should do is move the CTRL+C handler to move to the fast exit path.

Youssef1313 · 2025-03-23T08:57:35Z

@Evangelink There is also one thing I saw in MSBuild when I was looking recently:

https://github.com/dotnet/msbuild/blob/88496b9c8a4e9b4a84bee8889e47cf106dd7d96d/src/MSBuild/XMake.cs#L1146-L1150

nohwnd · 2025-03-25T08:37:54Z

Summarized in internal document. let's discuss there, it has code samples and the whole plan, but is not short. I am happy to summarize it here once we agree there. Or post it as whole, there is no secret in it.

I think control break is interesting as an abort option, but personally I don't even know where Break is on my keyboard 😁

nohwnd · 2025-03-25T08:43:11Z

We want a fast exit, in which case we should end the process (Env fail fast for example) BUT we cannot guarantee cleanup.

This is what VSTest does under VS, and in console, even though there is a cancel gesture, and I don't think it is right. We should give the test framework a chance to do at least a minimal cleanup and chance to the plugins to finalize their work.

I would like the cancellation to become the norm (e.g. thanks to better reporting of the cancellation process), and abort without cleanup being the last resort. And if we now implement ctrl+c to kill the process, then there is no way back.

Unblock run when cancellation was requested but there are still runni…

024c3d8

…ng tests

nohwnd commented Mar 19, 2025

View reviewed changes

nohwnd added 2 commits March 19, 2025 16:41

Fix test to test cancellation correctly

0e7051c

what to do ?

a4afd44

nohwnd commented Mar 19, 2025

View reviewed changes

nohwnd added 3 commits March 19, 2025 17:50

Will continue later

c422b39

Yield so we can abandon

77355ac

Yield so we can abandon

3ed9e8f

Youssef1313 reviewed Mar 20, 2025

View reviewed changes

Evangelink reviewed Mar 21, 2025

View reviewed changes

Youssef1313 reviewed Mar 21, 2025

View reviewed changes

nohwnd mentioned this pull request Mar 31, 2025

CTRL+C does not complete the test run when there is long running test #5345

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unblock run when cancellation was requested #5267

Unblock run when cancellation was requested #5267

nohwnd commented Mar 19, 2025

nohwnd Mar 19, 2025

nohwnd Mar 19, 2025

nohwnd Mar 19, 2025

nohwnd Mar 19, 2025

Youssef1313 Mar 19, 2025

nohwnd Mar 19, 2025 •

edited

Loading

Youssef1313 Mar 19, 2025

nohwnd Mar 19, 2025

Youssef1313 Mar 20, 2025

nohwnd Mar 21, 2025

nohwnd Mar 21, 2025

Youssef1313 Mar 20, 2025

Evangelink Mar 20, 2025

Youssef1313 Mar 21, 2025

MarcoRossignoli Mar 22, 2025 •

edited

Loading

Evangelink commented Mar 23, 2025

Youssef1313 commented Mar 23, 2025

nohwnd commented Mar 25, 2025

nohwnd commented Mar 25, 2025

		string fileCreationPath = Path.Combine(testHost.DirectoryName, "fileCreation");
		File.WriteAllText(fileCreationPath, string.Empty);


		testHostResult.AssertOutputMatchesRegex("Canceling the test session.*");

Unblock run when cancellation was requested #5267

Are you sure you want to change the base?

Unblock run when cancellation was requested #5267

Conversation

nohwnd commented Mar 19, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nohwnd Mar 19, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoRossignoli Mar 22, 2025 • edited Loading

Choose a reason for hiding this comment

Evangelink commented Mar 23, 2025

Youssef1313 commented Mar 23, 2025

nohwnd commented Mar 25, 2025

nohwnd commented Mar 25, 2025

nohwnd Mar 19, 2025 •

edited

Loading

MarcoRossignoli Mar 22, 2025 •

edited

Loading