Repository navigation
[Interpreter] Raise a pending thread abort again after a catch handler - #135520
pcshrosbree wants to merge 5 commits into
Conversation
COMPlusCheckForAbort expects its caller to set up Thread::m_OSContext for the place where the execution resumes; the interpreter did not, so ReadyForAsyncException asserted. The abort was also dispatched from inside the C++ catch block that handles ResumeAfterCatchException, so when a frame of the same InterpExecMethod caught it, the next ResumeAfterCatchException was not caught. Do the check at the top of the try block instead, with m_OSContext describing the interpreter frame at the resume IP. Add a regression test and enable ControlledExecutionTests.CancelInTryAndExitCatchNormally again. Fixes dotnet#135463
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
|
Tagging subscribers to this area: @JulieLeeMSFT, @BrzVlad, @janvorli |
It aborts threads in a few scenarios and is a niche case.
| INSTALL_UNWIND_AND_CONTINUE_HANDLER; | ||
|
|
||
| if (resumedAfterCatch) | ||
| { |
There was a problem hiding this comment.
Ideally, we should have no special checks on this execution path and keep it as simple as possible, just so we can have the best perf possible, even if it is just a var init and a check for it.
In the context of this PR, isn't this implementation basically an implicit rethrow of the abort exception ? Sounds like we could avoid resuming altogether and throw the abort exception earlier, while executing the catch from EH, same as a rethrow I believe is processed. Maybe we could add a check for thread abort somewhere around INTOP_LEAVE_CATCH. cc @janvorli
There was a problem hiding this comment.
Agreed on the principle, and it was a little worse than one init and one test per entry: EXIT_FRAME re-enters the try through goto MAIN_LOOP on every return from a non-recursive interpreted call, so the flag was tested about once per interpreted call, not once per entry. 35f8910 takes it off that path entirely. To answer the second half properly, on Checked linux-x64 I also prototyped the INTOP_LEAVE_CATCH version and ran both through the same tests, so here is what I found.
Why it is not a rethrow. A rethrow re-raises the caught exception from inside the handler while its dispatch is still in flight. The abort re-raise is defined to happen after the catch has completed and its EH state is gone, at the resume point: CallCatchFunclet calls the funclet, then ExInfo::PopExInfos, then ResumeAfterCatch, and COMPlusCheckForAbort relies on that order. For a non-rude abort it returns early while GetThrowable() is non-null, which it is for the whole catch funclet, and ReadyForAbort rejects a context classified inside a catch handler on the aborting thread itself (TAStackCrawlCallBackWorker skips non-finally clauses only when the aborter is another thread). So a check at INTOP_LEAVE_CATCH that calls the existing helper never fires: with a trace line at the raise site it raised zero times in every run, and CancelInTryAndExitCatchNormally fails because the code after the catch runs.
Making it fire needs the parent InterpreterFrame plumbed into ExceptionClauseArgs for the resume-point context, a variant of the check without the pending-throwable clause, and a different walk start frame (the funclet's frames sit below the resume SP, so the existing computation lands on the owning InterpreterFrame and StackFrameIterator::Init asserts). With all of that the abort does get raised from the funclet, and then it is caught by the same catch again: the leavecatch island sits at the try's end offset, the new first pass starts from there, the handler runs again, raises again, and the native stack overflows after about four hundred nested dispatches. The same happens if the check is reduced to IsAbortRequested(). Where an enclosing try exists, its catch takes the abort first; the shape, with the traces as measured:
try
{
try { RunInfiniteLoop(); }
catch (ThreadAbortException) { s_trace += "inner catch "; goto Done; }
}
catch (ThreadAbortException) { s_trace += "outer catch "; }
Done:
s_after = true;| JIT (tiered) | flag at the try top (previous revision) | check on the resume path (this revision) | raise at INTOP_LEAVE_CATCH |
|
|---|---|---|---|---|
| trace | inner catch |
inner catch |
inner catch |
inner catch outer catch outer catch ... until the watchdog fires |
This revision matches the tiered JIT in that example, but other JIT configurations also differ from each other there (with tiering off the outer catch runs), so I would not call it a universal compiled-code comparison. What does hold from the code is that raising before the funclet returns bypasses the handler's normal post-return callbacks and keeps each re-raise nested on the native stack, and the loop above shows the dispatch has no notion of a completed handler to skip. I favour keeping the check at the resume point; the funclet prototype and its plumbing are on hand if you want to look at that route anyway.
What this revision does instead. The catch block for ResumeAfterCatchException only records the resume SP and IP and jumps to a label outside any try, which does the unwind as before, describes the resume point in m_OSContext, and calls the NOTHROW COMPlusCheckForAbort. If the abort is due it jumps to a second, cold try with its own dispatcher and unwind handler, calling DispatchManagedException(kThreadAbortException); that block's catch funnels back into the same label, so a further catch in the same InterpExecMethod that swallows the abort again is handled the same way. The resumedAfterCatch local and its test are gone; the main loop's entry and return paths are what they were before the first revision of this PR. The abort is still raised at the resume point after the EH state has been popped, where EECodeManager::ResumeAfterCatch raises it for compiled code.
Measured on Checked runtimes: test135463's three cases pass under the JIT and DOTNET_InterpMode 1, 2 and 3 on linux-x64, osx-arm64 and windows-x64, and ControlledExecutionTests passes under modes 1, 2 and 3 on linux-x64, 1 and 3 on osx-arm64 and 1 on windows-x64, with Linux interpreter marker counts confirming the new raise site. On linux-x64, in interpreter modes 1 and 3, nine abort shapes (the one above, catch inside finally, catch then finally, abort requested during a catch, a filter on an enclosing catch, consecutive trys, the stack-trace shape, and the test's three) give identical traces for the previous revision and this one in every cell. On the entry-path cost, in a linux-x64 Release build the instruction path from an interpreted return back to the first opcode dispatch goes from 37 to 28 instructions, and the function-entry path is unchanged within one instruction (register allocation moved); on linux-x64 Checked and Release, InterpMode 1 with tiering disabled, a 20M-iteration interpreted call/return loop and a 2M-iteration native-to-interpreter entry loop show median differences between the two revisions within the run-to-run spread, so I would not claim a measurable win, only that nothing abort-specific executes there any more.
One thing the comparison turned up about the test itself: under InterpMode 2 and 3 the s_trace += "unreachable " after each catch is an interpreted String.Concat, and a runtime that never re-raised at the end of the catch still passed those three cases there, consistent with a safepoint of that call raising the pending abort before the store (measured with the check removed, Checked linux-x64; the same runtime fails under mode 1 and in ControlledExecutionTests). 7e70fe0 records that point with a call-free store and asserts on it; with the re-raise removed the test now fails in all tested interpreter configurations on Checked linux-x64.
For completeness I also tried raising from InterpreterCodeManager::ResumeAfterCatch directly, mirroring EECodeManager: the interpreter's frame chain is unwound only in InterpExecMethod's catch, so the new first pass starts at the original throw site, the same catch handles the abort again, and CatchInsideOfFinally runs its finally twice.
Note
This reply and the two commits were prepared with AI assistance (Anthropic Claude, checked by OpenAI Codex) at my request. I reviewed the change and the measurements before posting.
There was a problem hiding this comment.
A clarification of the LeaveOutOfOuterTry comparison in my previous comment, where this revision matched the tiered JIT and I noted that the JIT configurations differ from each other there. The match with tier-0 code was real, but that tier-0 result comes from a JIT issue I have since reported as #135637, with a proposed fix in #135638: in tier 0 the block the catch returns to generates no code, so the abort is dispatched as if it had been raised in the code after the enclosing try. With #135638, tier 0, DOTNET_JITMinOpts=1 and DOTNET_TieredCompilation=0 all give inner outer. That is also the behaviour the JIT intends: when a catch's leave exits an enclosing try that has a catch, impImportLeave makes the catch return to a step block inside that try, and its comment says why (importer.cpp):
If a try/catch catches a ThreadAbortException (which might be because it catches a parent, e.g. System.Exception), and the catch doesn't call System.Threading.Thread::ResetAbort(), then when the catch returns to the VM, the VM will automatically re-raise the ThreadAbortException. When it does this, it uses the target address of the catch return as the new exception address. That is, the re-raised exception appears to occur at the catch return address. If this exception return address skips an enclosing try/catch that catches ThreadAbortException, then the enclosing try/catch will not catch the exception, as it should.
[...]
Note that this pattern isn't theoretical: it occurs in ASP.NET, in IL code generated by the Roslyn C# compiler.
Compared with the JIT built with #135638, the interpreter with this PR differs on that shape: inner under DOTNET_InterpMode=1 and 3, where the JIT gives inner outer (Checked linux-x64; builds of this PR's runtime change at 35f8910 and at the previous revision 08e3dfb, each with a trace line added at the raise site; three runs per mode and build, all identical). The other two shapes of #135637 match the fixed JIT (catch finally outer, with the finally run, and inner).
From the source and an InterpDump, the difference comes from where the interpreter compiler sends the catch, not from this PR's raise. CreateLeaveChainIslandBasicBlocks creates a leave island for each catch handler and each finally-protected try that a leave exits, and none for an enclosing try protected by a catch, so the inner catch's leavecatch goes straight to the leave target, past the outer try. DOTNET_InterpDump=LeaveOutOfOuterTry with DOTNET_InterpMode=1 and this PR (excerpt, comments mine):
0018: IR_0004: call [0 <- 16], .AbortReraise:RunInfiniteLoop()
0028: IR_0008: br [nil <- nil], IR_000c
0030: IR_000a: leavecatch [nil <- nil], IR_0010 ; end of the inner catch
0038: IR_000c: br [nil <- nil], IR_0010
0040: IR_000e: leavecatch [nil <- nil], IR_0010 ; end of the outer catch
0048: IR_0010: ldc.i4 [0 <- nil], 1 ; Done:
...
try [IR_0004(0), IR_000a(7)) catch handler [IR_001c(7), IR_0035(1e))
try [IR_0004(0), IR_000e(20)) catch handler [IR_0035(20), IR_004e(37))
The outer try is [IR_0004, IR_000e). The inner catch resumes at IR_0010, and the instruction just before that is the outer catch's leavecatch at IR_000e, already outside the range, so for this shape the outer catch is skipped whether handlers are looked up at the resume address or just before it.
I don't think this should hold up this PR, and I don't know whether the interpreter means to follow the JIT here. If it does, a step island inside the enclosing try for the catch to return to, as the JIT has, looks like the natural shape. I'm happy to open an issue for it or try a change, whichever you prefer. Until then the regression test in #135638 runs on jitted code only, for this reason.
Note
This reply was prepared with AI assistance (Anthropic Claude, checked by OpenAI Codex) at my request. I reviewed it before posting.
The check that raises a pending thread abort again after an interpreted catch handler ran at the top of InterpExecMethod's try block, behind a flag set when a frame resumed. That try block is entered on every interpreter entry and on every return from a non-recursive interpreted call (EXIT_FRAME jumps back to MAIN_LOOP), so the flag was tested about once per interpreted call. Move the check back to the resume path. The catch block for ResumeAfterCatchException only records the resume SP and IP and jumps to RESUME_AFTER_CATCH, outside any try block, which unwinds the interpreter frames, describes the resume point in Thread::m_OSContext and calls COMPlusCheckForAbort, which does not throw. If the abort is due, it jumps to ABORT_AFTER_CATCH, a try block of its own that dispatches the ThreadAbortException; its catch funnels back into RESUME_AFTER_CATCH, so a further catch in the same InterpExecMethod that swallows the abort again is handled the same way. The entry and return paths are what they were before the check was added, and the abort is still raised at the resume point after the EH state has been popped, as EECodeManager::ResumeAfterCatch does for compiled code.
The three cases recorded "reached the code after the catch" with s_trace += "unreachable ", which is a call to String.Concat. Under DOTNET_InterpMode 2 and 3 the cases passed on a runtime that did not raise the abort again at the end of the catch, consistent with a safepoint of that call raising the pending abort before the store. Replace the store with a volatile bool set without a call and assert that it stayed false; with the re-raise removed the test now fails in all tested interpreter configurations on a Checked linux-x64 runtime.
…#135531) Fixes #135530. On windows-x64 with shadow stacks, a Checked runtime asserts in `CallCatchFunclet` when a thread abort that the runtime raises again at the end of a catch handler is caught by another handler in the same method. `EECodeManager::ResumeAfterCatch` sends such a thread to `ThrowControlForThread`, and `RedirectForThrowControl` records in the `FaultingExceptionFrame` the SSP at the point where the thread was redirected. That frame was interrupted rather than stopped at a call, so the slot below its SSP does not hold its `ControlPC`, and the assert's last clause fails when the catch handler is in that frame. The SSP points at the frame's return address, which is what resuming the frame needs. The assert now also accepts a handler frame that has faulted. Release builds are unchanged. Test: new `src/tests/Regressions/coreclr/GitHub_135530`, the case from the issue. It is disabled under the interpreter until #135520 (fixes #135463) is in; the interpreter fails it for that reason, not this one. Validation (Checked): | | main | this PR | |---|---|---| | windows-x64 (CET enabled), JIT | assert in `CallCatchFunclet` | pass | | windows-x64, JIT with `TieredCompilation=0`, `JITMinOpts`, `JitStress` 1 and 2, `ReadyToRun=0`, `GCStress` 0x3 and 0xC | | pass | | linux-x64 and osx-arm64, JIT, same configurations | pass | pass (the assert is Windows x64 only) | The other probes from the issue investigation also print the same on windows-x64 as on linux-x64 with this change, including an abort raised again right in front of a try block that catches it. > [!NOTE] > This PR was prepared with AI assistance (Anthropic Claude, reviewed by OpenAI Codex). The builds and runs it cites were done at the PR's base commit.
`GC/API/GC/GetGeneration` fails intermittently on CoreCLR legs because an object allocated just before `GC.Collect()` is still in generation 0 afterwards: ```text Running test: GC/API/GC/GetGeneration/GetGeneration.dll ObjectTest Passed! 0 0 arrayTest Failed! failTest Passed! Test for GetGeneration() FAILED! ``` #126108 removed this test's `RequiresProcessIsolation`, which had been there for `GCStressIncompatible`; that skip became a `[SkipOnCoreClr]` attribute. Since then the test has run in-process in the merged `GC` runner, after other tests. #132076 restored isolation for `Collect`, `Collect0` and `Collect1`, whose generation assertions had started failing after the same change. It did not include `GetGeneration`. In #132291, janvorli said the most likely reason is the same as for the Collect tests and that the fix would be the same as #132076. This PR adds that change to `GetGeneration.csproj`: the same property group, with the same comment. I chose not to make the test tolerate an object that stays in gen0. `objectTest` and `arrayTest` exist to check that a full collection promotes a live, newly allocated object, so accepting "0 0" would remove the check rather than fix it. The other tests in the directory that call `GetGeneration` (`GetGenerationWR`, `GetGenerationWR2`, `GetGeneration_box`, `GetGeneration_fail`) do not assert promotion, so I left them in the merged runner. Validation used one existing local linux-x64 Checked runtime, with the priority-0 GC test tree built at this PR's parent and at this commit. The runtime was the control build from the earlier #135520 investigation, not an unmodified main build; it was not rebuilt at either of these commits, since this change touches only a test project. - Before the change, the merged runner runs `GC/API/GC/GetGeneration/GetGeneration.dll` in-process. After it, the runner reports `GC/API/GC/GetGeneration/GetGeneration.cmd` and runs the generated `GetGeneration.sh` in a child process, as it already does for `Collect0` and `Collect1`. The test's result name in CI changes accordingly, from `.dll` to `.cmd`. - I ran the full merged `GC` runner from each tree 200 times, interleaved, with 12 runs in parallel and the same corerun arguments as the plain CI work item reported in #132291. All 53 tests passed in every run of both sets. Isolation adds one test process per invocation. The median runner time was 98.25 s with the change and 97.65 s without; these concurrent local timings do not establish its CI cost. These runs do not demonstrate a reduction in failures. The earlier investigation described in my comment on #132291 recorded one `GetGeneration` failure in 400 full merged-runner runs, 200 on each of two runtime builds used to investigate #135520, both without this isolation change. - To check the mechanism, I loaded the merged-runner `GetGeneration.dll` into a host process and called `TestEntryPoint` 30,000 times (three runs of 10,000), resetting the test's static counter between calls. Before each call, the host pinned 200 arrays of 1,000 bytes, ran one collection while they were held, released them, and pinned 200 new ones. 56 of the calls failed, and the first failure in each run printed `ObjectTest Passed!`, `0 0`, `arrayTest Failed!`, as in CI. The same host without the pins had no failures in 30,000 calls, and pinning only right before each call had none in 6,000. This shows one way that pins and prior collections in the same process can produce this failure; it does not establish whether pinned objects caused the CI failures or where they came from. With the change, the test runs in its own process, which pins held by the runner process cannot affect. Fixes #132291 cc @kkokosa, since #132291 is assigned to you. > [!NOTE] > This change was prepared with AI assistance (Anthropic Claude, checked by OpenAI Codex) at my request. I reviewed the change and the measurements before opening.
|
@pcshrosbree I guess I can see the problem with the leave island you have shown. I think my original idea was that the catch leave island would be the resume after catch target. So the catch would end with INTOP_LEAVE_CATCH pointing to that island and that island would contain plain INTOP_BR. Then we would resume after catch there and rethrowing of the thread abort exception from that place would still be covered by the right try block. |
|
@janvorli @BrzVlad, the flip gives the layout you describe. On its own, though, it makes this PR loop, so the abort raise has to change with it. I prototyped two ways to do that. Which do you want: raise at the resume point, or raise at
I prototyped all three on Checked runtimes on linux-x64, osx-arm64 and windows-x64. Sizes exclude trace lines. Two other observations relevant to the choice:
The flipIn both test cases from #135871, the catch now ends with Why the flip alone loopsWith the flip alone, test135463 hangs in all six of its interpreter configurations (the seventh runs under the JIT), 8 of 10 abort test cases loop, and one more runs a finally twice, on all three platforms. My reading of the code, consistent with the traces (I did not trace the lookup itself): the resume path clears the frame's faulting flag, so handlers are looked up at the instruction before the island. When the island comes right after a try body, that instruction is the try's last one, so the same catch takes the abort again. Marking the frame faulting before the raise, which is what the resume-point option adds, removes the loops. What the
|
| <CLRTestPriority>1</CLRTestPriority> | ||
| <!-- Needed for CLRTestTargetUnsupported --> | ||
| <RequiresProcessIsolation>true</RequiresProcessIsolation> | ||
| <!-- ControlledExecution is supported on CoreCLR only --> |
ControlledExecution is supported on CoreCLR only. Skip the test with PlatformDetection.IsCoreCLR, which is false on Mono and NativeAOT, instead of CLRTestTargetUnsupported and NativeAotIncompatible. Neither is set any more, so the test no longer needs its own process and runs in the merged Regressions runner: the abort is confined to ControlledExecution.Run, which resets it before returning.
|
I agree with @BrzVlad that it is hard to reason about a combination of changes. Maybe you could create a new PR with the flip and extra changes needed with that so we can easily see it in its completeness. And I've realized that re-raising thread abort should really happen at the resume point, so that it behaves like JITted code. After the flip, the |
|
@janvorli, @BrzVlad: #135939 has the whole change in one commit based on main: the flip, the abort raised again at the resume point with the frame marked faulting, and the tests, including both cases from #135871. I'll close this PR in its favour. I had also prepared the two versions side by side as draft PRs, one raising the abort at the resume point and one at the |
Fixes #135463.
When interpreted code catches a
ThreadAbortException, the abort has to be raised again at the end of the catch handler. The interpreter does that in theResumeAfterCatchExceptionhandler ofInterpExecMethod, and it had two problems there:COMPlusCheckForAbortwithout setting upThread::m_OSContext, whichCOMPlusCheckForAbortleaves to its caller.Thread::ReadyForAsyncExceptionasserts that the context points at managed code and walks the stack from it.EECodeManager::ResumeAfterCatchsets it up for compiled code. Now the interpreter fills it from the interpreter frame at the resume IP, withInterpreterFrame::SetContextToInterpMethodContextFrame.catchblock. When a frame of the sameInterpExecMethodcatches that abort, theResumeAfterCatchExceptionthat resumes it is thrown while the catch block is still active, so thetryblock does not catch it (Checked: assertpFrame->PtrNextFrame() == NULLin exceptionhandling.cpp). Now the catch block sets a flag and the check runs at the top of thetryblock.The abort is still raised as a return address at the resume IP, as before, so a try block that starts right at the resume point does not cover it. That matches optimized JIT code, which resumes on a
nopin front of such a try block.Tests:
src/tests/Regressions/coreclr/GitHub_135463: the abort caught in the method, caught again by its caller, and kept pending at the end of a catch inside a finally. It checks the exception and which blocks ran.ControlledExecutionTests.CancelInTryAndExitCatchNormallyis enabled again.CancelItselfOutsideOfTryCatchFinallystays disabled: underDOTNET_InterpMode=1it fails for a different reason, an abort that compiled code raises again throughThrowControlForThreadand that is then resumed in an interpreted frame. I'll open a separate issue for that.Validation on linux-x64, osx-arm64 and windows-x64 (Checked):
DOTNET_InterpMode1/2/3, with and withoutTieredCompilation=0ReadyForAsyncExceptionControlledExecutionTests(9 tests), JITControlledExecutionTests(9 tests),DOTNET_InterpMode=1On windows-x64 I also ran
ControlledExecutionTestsone test per process underDOTNET_InterpMode2 and 3, with a local workaround for #135535, which otherwise stops the tests during discovery. On main,CancelInTryAndExitCatchNormallyhits the same assert. With this PR, all 9 pass.The new test also passes under the JIT with
TieredCompilation=0,JITMinOpts,JitStress1 and 2,ReadyToRun=0, andGCStress0x3 and 0xC. Each part of the change was taken out in turn and the test fails under the interpreter every time: without them_OSContextsetup it hits the assert above, without recording the resume IP it hangs, and with the check left in the catch block it hits thePtrNextFrameassert.Note
This PR was prepared with AI assistance (Anthropic Claude, reviewed by OpenAI Codex). The builds and runs it cites were done at the PR's base commit.