Exclude reverse P/Invoke stubs from profiler code-transition callbacks.

[lateralusX]

Summary

Reverse P/Invoke marshaling stubs emit a spurious profiler code-transition callback pair around the call to their managed target. This change stops emitting it, making reverse P/Invoke consistent with reverse COM and reverse delegate stubs, and adds a regression test for #120151.

Background

ILStubState::FinishEmit wraps a stub's call to its target with ManagedToUnmanagedTransition / UnmanagedToManagedTransition profiler callbacks. That is correct for forward stubs, where the target is native
and the call genuinely leaves the runtime.

A reverse P/Invoke stub, however, runs in managed code and just calls the managed target — no native boundary is crossed at that point. The callback pair is therefore spurious, and because the emit path is forward-templated it is also
reported in the wrong direction (M2U(CALL) … U2M(RETURN)). The real unmanaged↔managed transition for a reverse P/Invoke is already reported at the frame boundary (JIT_ReversePInvokeEnter/Exit, and the prestub/interpreter reverse paths) with the correct direction and the real target MethodDesc.

Reverse COM and reverse delegate stubs were already excluded from this path. Reverse P/Invoke was the only reverse stub still going through it. Historically it reported a bogus, non-null FunctionID here (#120151); that value was later neutralized to NULL, but the spurious, wrong-direction transition itself remained.

Change

Gate the transition-callback emission in FinishEmit on SF_IsForwardStub, so reverse P/Invoke stubs no longer emit the phantom pair — the same treatment reverse COM/delegate stubs already receive. This also restores the SF_IsForwardStub assumption asserted in EmitProfilerEndTransitionCallback (previously stale for the reverse P/Invoke path).

Profiler behavior change

This is profiler-observable: a code-transition callback that previously incorrectly fired (with a NULL FunctionID in .NET11 and corrupted pointer in .NET 8/9/10) for reverse P/Invoke stubs no longer fires. The authoritative frame-boundary transition for the reverse call is unchanged, so profilers still see the real unmanaged↔managed transition with the correct
FunctionID.

Regression test

Adds an IJW (C++/CLI) profiler test for #120151 that drives the managed → native → managed-by-pointer scenario through a reverse marshaling stub. It asserts the target is reported CALL in / RETURN out and that no nested code transitions occur inside the target (previously the stub emitted a NULL-FunctionID pair here — and, before the original fix, a bogus non-null one). Every observed transition is traced to the test log for inspection.

Fixes #120151

[azure-pipelines]

Azure Pipelines:
Successfully started running 4 pipeline(s).
12 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @tommcdon, @dotnet/dotnet-diag
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

Adds a Windows-only IJW (C++/CLI) profiler regression test to validate code-transition callback behavior when a managed method is invoked via a raw native function pointer, along with a new native profiler implementation used by the test.

Changes:

• Added a new IjwProfiler implementation under the shared test profiler DLL to validate transition FunctionID validity and expected transition shape.
• Added a new IJW profilee mixed-mode DLL (IjwProfileeDll) plus managed harness (ijw.csproj / ijw.cs) to drive the repro scenario.
• Wired the new profiler into the native profiler build (CMakeLists.txt) and COM class factory.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/tests/profiler/native/ijw/ijwprofiler.h	Defines the new IJW-focused profiler used by the regression test.
src/tests/profiler/native/ijw/ijwprofiler.cpp	Implements transition validation logic and pass/fail reporting for the profiler test.
src/tests/profiler/native/CMakeLists.txt	Adds the new IJW profiler implementation to the shared profiler DLL build.
src/tests/profiler/native/classfactory.cpp	Registers the new profiler CLSID so it can be instantiated by the test runner.
src/tests/profiler/ijw/IjwProfileeDll.cpp	Mixed-mode IJW profilee implementing the managed→native→managed-by-pointer call shape.
src/tests/profiler/ijw/ijw.csproj	New Windows-only profiler test project that builds the native pieces and runs under ProfilerTestRunner.
src/tests/profiler/ijw/ijw.cs	Managed harness that launches the profiled process and triggers the IJW-by-pointer scenario.
src/tests/profiler/ijw/CMakeLists.txt	CMake project for building/installing the IJW mixed-mode profilee DLL.

Comments suppressed due to low confidence (2)

src/tests/profiler/native/ijw/ijwprofiler.h:44

• Track whether any nested transitions were observed (and whether they were NULL FunctionID) so Shutdown() can assert the contract was exercised, not just that no non-null value was seen incidentally.

private:
    std::atomic<int> _failures;
    std::atomic<int> _transitions;

src/tests/profiler/native/ijw/ijwprofiler.cpp:124

• The nested-transition check only fails on non-null FunctionID, but it doesn’t record that a nested transition was actually seen (and that it was NULL). Incrementing counters here lets Shutdown() assert the contract was exercised and reduces the chance of a false pass if the runtime stops reporting these transitions.

    // Any non-target transition seen while executing the managed target is a
    // nested reverse (unmanaged->managed) marshaling stub. Post-fix these report
    // a NULL FunctionID; a non-null value here is exactly the 120151 bug (a
    // resolvable-but-wrong pointer would slip past the check above).
    if (_insideTarget.load() && functionID != 0)
    {
        _failures++;
        printf("FAIL: nested %s inside target reported non-null FunctionID=0x%p (reason=%d)\n",
               which, (void*)functionID, (int)reason);
        fflush(stdout);
    }

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/tests/profiler/ijw/ijw.csproj:8

• is already set for all profiler tests via src/tests/profiler/Directory.Build.targets, so the project-local property (and the comment claiming it’s needed for CMakeProjectReference) is redundant and potentially misleading.

    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
    <!-- Needed for CMakeProjectReference -->
    <RequiresProcessIsolation>true</RequiresProcessIsolation>
    <CrossGenTest>false</CrossGenTest>

src/tests/profiler/ijw/ijw.cs:23

• The reflection-based load path can throw or null-ref with little context (e.g., GetType/GetMethod returning null). Adding explicit error messages here makes failures actionable if the IJW profilee DLL isn’t found or is missing the expected type/method.

            Assembly ijwDll = Assembly.Load("IjwProfileeDll");
            Type testType = ijwDll.GetType("TestClass");
            object testInstance = Activator.CreateInstance(testType);
            MethodInfo method = testType.GetMethod("CallManagedFunctionByPointer");
            return (int)method.Invoke(testInstance, null);

src/tests/profiler/native/ijw/ijwprofiler.cpp:23

• IjwProfiler::Initialize assumes pCorProfilerInfo is non-null after Profiler::Initialize, but Profiler::Initialize returns S_OK even if QI fails and can leave pCorProfilerInfo == nullptr. This can lead to a null dereference when calling SetEventMask2, obscuring the actual failure mode.

    Profiler::Initialize(pICorProfilerInfoUnk);

    HRESULT hr = S_OK;
    if (FAILED(hr = pCorProfilerInfo->SetEventMask2(COR_PRF_MONITOR_CODE_TRANSITIONS | COR_PRF_DISABLE_INLINING, 0)))
    {

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/tests/profiler/ijw/IjwProfileeDll.cpp:17

• The profilee depends on the compiler's per-function /clr defaults to get the required repro shape (native helper calling a managed target via raw function pointer). Most existing IJW tests explicitly use #pragma managed / #pragma unmanaged to avoid this being a compiler-version-dependent detail; without explicit pragmas this test could silently stop exercising the reverse-stub path (e.g., if call or ManagedByPointerTarget ends up compiled in the wrong mode). Consider making the managed/unmanaged boundaries explicit here and updating the comment accordingly.

// Compiled with default /clr per-function codegen (no #pragma managed/unmanaged);
// the reporter's sample used a file-static function, but a named function
// reproduces identically and lets the profiler match it by name.
__declspec(noinline) void ManagedByPointerTarget() {}
__declspec(noinline) static void call(void (*f)()) { f(); }

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/coreclr/vm/dllimport.cpp:656

• The PR description says the fix was already done in #117901 and that the test should validate nested reverse-stub transitions report a NULL FunctionID. However this PR also changes CoreCLR stub generation to only emit profiler transition callbacks for forward stubs (SF_IsForwardStub), which removes reverse-stub transition callbacks entirely. Please either (a) update the PR description to mention this additional runtime behavior change and why it’s required, or (b) drop/move this runtime change into the PR that actually fixes the behavior being tested so the scope matches the title/description.

        // Notify the profiler of call out of the runtime
        if (CORProfilerTrackTransitions()
            && !SF_SkipTransitionNotify(m_dwStubFlags)
            && SF_IsForwardStub(m_dwStubFlags))

src/tests/profiler/ijw/ijw.cs:23

• Activator.CreateInstance(Type) can return null (for example, if the type has no public parameterless ctor). If that happens, the subsequent MethodInfo.Invoke will fail with a less actionable exception. Consider throwing a clear InvalidOperationException when instance creation fails.

                ?? throw new InvalidOperationException("Could not find type 'TestClass' in IjwProfileeDll.");
            object testInstance = Activator.CreateInstance(testType);
            MethodInfo method = testType.GetMethod("CallManagedFunctionByPointer")