System.Private.Xml.Tests crashing with SIGSEV

[ericstj]

Failing test -- https://helixr18s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-128864-merge-de83ef25c2f0420394/System.Private.Xml.Tests/1/console.8950616e.log?helixlogtype=result

Dump https://helixr18s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-128864-merge-de83ef25c2f0420394/System.Private.Xml.Tests/1/core.1000.17?helixlogtype=result

I couldn't analyze that dump - but it could be an environment issue (this is a linux arm64 dump, I have x64 - I tried windows and linux and both failed to load SOS). I also noticed that the build analysis infra that's supposed to auto-analyze and dump to log wasn't working cc @hoyosjs

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1445922
Build error leg or test failing: System.Private.Xml.Tests.WorkItemExecution
Pull request: #128864

Error Message

DO NOT USE JSON BELOW IF THIS IS A BUILD BREAK otherwise build analysis will allow pull requests to merge that break the build worse. For a build break, do not use this issue form. Make a regular new issue.

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["xunit.console.dll System.Private.Xml.Tests.dll", "exit code 139"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1445922
Error message validated: [xunit.console.dll System.Private.Xml.Tests.dll exit code 139]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/3/2026 4:06:49 PM UTC

Report

Build	Repository	Test	Pull Request
1488812	dotnet/runtime	System.Private.Xml.Tests.WorkItemExecution	#130058
1483366	dotnet/runtime	System.Private.Xml.Tests.WorkItemExecution	#129255
1453333	dotnet/runtime	System.Private.Xml.Tests.WorkItemExecution	#129110

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	3

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

[mangod9]

Issue #128953 — System.Private.Xml.Tests SIGSEGV (Mono MiniJIT, Linux ARM64)

Summary

SIGSEGV (exit code 139) in System.Private.Xml.Tests on Linux ARM64 Mono MiniJIT (Debug build). The crash occurs in JIT-compiled code that attempts to execute a safepoint poll through a thunk pointing to unmapped memory. The crash reporter itself then crashes in _Unwind_Backtrace, making the core dump harder to analyze.

Environment

• Platform: Linux ARM64 (Azure Linux 3, arm64v8 Docker container on Helix)
• Runtime: Mono MiniJIT (Debug), .NET 11.0.0-preview
• Build: 1445922
• PR: #128864 (infrastructure-only change — Move PlatformDocAnalyzer to Analyzers.targets)
• Helix leg: build_linux_arm64_debug_mono_minijit_librariestests

Dump Analysis

Core dump source

• /tmp/dump-128953/core.1000.17 — ELF 64-bit ARM64 Linux core file (209MB)
• Runtime symbols from Helix correlation payload

The Double-Crash Problem

The core dump captures a secondary crash, not the original one:

	Original SIGSEGV	Secondary SIGSEGV (in core)
Where	JIT-compiled safepoint poll	_Unwind_Backtrace in libgcc_s.so.1
si_code	1 (SEGV_MAPERR)	0 (SI_USER)
Fault addr	0xf300ffbd12a0 (unmapped)	N/A (signal re-raised)
Cause	Branch to unmapped thunk target	Crash in backtrace() during crash reporting

The original SIGSEGV triggers Mono's crash handler (mono_sigsegv_signal_handler_debug → mono_handle_native_crash → mono_dump_native_crash_info), which calls backtrace() → _Unwind_Backtrace, and _Unwind_Backtrace itself crashes (with x24=0x26, an invalid pointer).

Original Crash — Saved Signal Context

From the ucontext_t on the signal handler's stack at 0xf2fff6ffb980:

Original SIGSEGV registers:
  PC       = 0xf300ffbd12a0   ← CPU tried to EXECUTE here (unmapped!)
  LR (x30) = 0xf30010cd0374   ← return address in JIT code
  SP       = 0xf2fff6ffcab0
  x16      = 0xf300ffbd12a0   ← indirect branch target register
  FP (x29) = 0xf2fff6ffcab0

PC = fault_address = x16: The crash was an indirect branch (br x16) to unmapped memory.

The Crashing Thunk

At LR 0xf30010cd0374, the calling instruction at 0xf30010cd0370 is bl 0xf30010cd1474 — a branch to a thunk:

0xf30010cd1474:  ldr x16, 0xf30010cd147c    ; load 64-bit target from literal pool
0xf30010cd1478:  br  x16                     ; branch to target
0xf30010cd147c:  .quad 0x0000f300ffbd12a0    ; BAD TARGET — unmapped memory!

Safepoint Poll Pattern

The JIT code at the crash site implements a GC safepoint poll:

0xf30010cd035c:  mov  x0, #0x7530                     ; x0 = &mono_polling_required
0xf30010cd0360:  movk x0, #0x1b46, lsl #16
0xf30010cd0364:  movk x0, #0xf300, lsl #32
0xf30010cd0368:  ldr  x17, [x0]                        ; x17 = mono_polling_required value
0xf30010cd036c:  cbz  x17, 0xf30010cd0374              ; skip if polling not required
0xf30010cd0370:  bl   0xf30010cd1474                    ; CALL THUNK → should be mono_threads_state_poll
0xf30010cd0374:  str  w19, [x29, #1016]                ; ← LR lands here after call

This pattern is emitted by OP_GC_SAFE_POINT in mini-arm64.c:5931-5940 via emit_call(MONO_JIT_ICALL_mono_threads_state_poll).

Thunk Table Analysis

The method's thunk area shows multiple thunks, some valid, some corrupted:

Thunk Address	Target	Status
0xf30010cd1444	0xf3001afe6c90	✅ Valid (Mono trampoline — lazy resolver)
0xf30010cd1454	0xf3001afe6c90	✅ Valid (same trampoline)
0xf30010cd1464	0xf300ffbd12a0	❌ BAD — unmapped
0xf30010cd1474	0xf300ffbd12a0	❌ BAD — crashing thunk
0xf30010cd1484	0xf300ffbd12a0	❌ BAD — same bad target
0xf30010cd1494+	0x0000000000000000	Free (unused)

Key observations:

• Thunks 0x1444/0x1454 still point to the lazy resolver trampoline at 0xf3001afe6c90
• Thunks 0x1464/0x1474/0x1484 were patched with 0xf300ffbd12a0 (3 different thunks, same bad target)
• mono_threads_state_poll should be at 0xf3001b3109a8 — neither address matches

Pre-Signal Stack Trace

The stack above the signal frame shows the trampoline resolution chain that ran on this thread:

mono_magic_trampoline (mini-trampolines.c:789)
  → common_call_trampoline (mini-trampolines.c:758)
    → mono_arch_patch_callsite (tramp-arm64.c:38)
      → arm_patch_full (arm64-codegen.h:176)
        → create_thunk (mini-arm64.c:1368)

This means the thunk was being patched (or had just been patched) by the trampoline resolution mechanism when the crash occurred.

Root Cause Analysis

How Thunks Work (ARM64)

ARM64 bl instructions can only reach ±128MB. When a call target is farther, the JIT creates a thunk in a reserved area adjacent to the method's code:

Thunk (16 bytes = THUNK_SIZE):
  ldr x16, [pc+8]      ; load 64-bit target from literal pool
  br  x16               ; indirect branch
  .quad <target_addr>   ; 8-byte target address

Lifecycle:

1. During JIT compilation: emit_call records a MONO_PATCH_INFO_JIT_ICALL_ID patch with MONO_R_ARM64_BL relocation
2. At link time: mono_resolve_patch_target resolves the icall → returns a lazy trampoline address (via mono_icall_get_wrapper)
3. arm_patch_full(cfg, ...) creates a thunk pointing to the lazy trampoline
4. At first call: trampoline fires → mono_magic_trampoline → common_call_trampoline compiles the icall wrapper → mono_arch_patch_callsite → arm_patch_full(NULL, ...) → create_thunk(NULL, ...) rewrites the thunk with the compiled wrapper's address

What Went Wrong

The thunk target 0xf300ffbd12a0 was patched by step 4 (confirmed by the trampoline resolution chain on the stack). This address is unmapped — it's ~3.8GB away from any mapped region in the process, consistent with a freed mmap allocation (a JIT code buffer that was munmap'd).

Primary Hypothesis: Stale Resolved Address from Freed Code Manager

mono_jit_free_method (mini-runtime.c:2986) frees dynamic method code but does NOT invalidate thunks in other methods.

System.Private.Xml heavily uses dynamic code generation (AssemblyBuilder/TypeBuilder/ILGenerator in CodeGenerator.cs:1284) for XML serialization. When a dynamic method is GC-collected:

1. mono_jit_free_method → mono_code_manager_destroy(ji->code_mp) (line 3083) → munmap of the code buffer
2. The freed code buffer's address becomes stale
3. jump_target_hash cleanup (lines 3042-3057) only removes entries pointing INTO the freed code — it does not scan all methods' thunk tables for thunks targeting the freed address

The critical gap: there is zero code in Mono to invalidate thunks that were patched to point to a freed method's address. Verified by searching for thunk_invalidat, free_thunk, thunk_free, invalidate_thunk across the entire mono/mini/ directory — no results.

The MONO_ARCH_HAVE_INVALIDATE_METHOD path (line 3060) only handles the special case of delegate wrappers with keep_delegates debug option.

Why the Safepoint Poll Is Affected

The safepoint poll calls mono_threads_state_poll through a JIT icall wrapper. The wrapper itself should be in a permanent code manager. However, the trampoline resolution flow in common_call_trampoline (line 628) calls mono_jit_compile_method(m, error) which may in some edge cases compile code into a collectible jit_mm context — this is exactly the class of bug fixed by PR #125583 (jit_mm mismatch in delegate trampolines for dynamic methods).

The address pattern strongly suggests a freed code buffer:

• 0xf300ffbd12a0 — the bad target
• 0xf3001b3109a8 — expected mono_threads_state_poll address
• 0xf3001afe6c90 — lazy resolver trampoline (still valid)
• The bad address shares the 0xf300 high bytes (same virtual address space region) but the lower bits differ wildly — consistent with a different mmap'd allocation in the same ASLR region

Supporting Evidence

1. PR [mono] Fix jit_mm mismatch in delegate trampoline creation for dynamic methods #125583 (March 2026): Fixed jit_mm mismatch in mono_create_delegate_trampoline_info for dynamic methods — proves this class of bug exists in the trampoline infrastructure
2. 3 of 6 thunks patched to the same bad address: Consistent with 3 safepoint polls being called (resolving through trampolines) while the dynamic assembly was alive, then the code being freed
3. 3 thunks still at lazy trampoline: The remaining safepoint polls in this method were never called, so their thunks were never updated from the original trampoline
4. Test is System.Private.Xml.Tests: XML serialization is one of the heaviest users of dynamic code generation in .NET

Alternative Hypotheses

1. Race condition in emit_thunk: While create_thunk holds jit_mm_lock, the 8-byte target write at *(guint64*)code = target is atomic on ARM64 (aligned 64-bit store). However, there's no memory barrier between the store and icache flush — another CPU could read the old value. This is unlikely to produce an UNMAPPED address (old value would be the trampoline, which is valid).

2. Memory corruption: A wild write corrupting the thunk's literal pool. Less likely because exactly 3 thunks have the same consistent bad value.

3. Code manager reuse: The freed code buffer's virtual address was reused by another mmap for different code, which was then also freed. This would explain why the address is completely unmapped.

[mangod9]

Update: Root Cause Identified — Cross-Cache-Line Torn 64-bit Access

Further investigation of the thunk alignment reveals the root cause of the +4GB corruption.

The Alignment Problem

All thunk literals in this method are 4-byte aligned but NOT 8-byte aligned. This is because the thunk area starts at code + code_size + unwindlen (mini.c:2146-2147), and code_size + unwindlen is not guaranteed to be a multiple of 8:

cfg->thunks_offset = cfg->code_size + unwindlen;
cfg->thunks = code + cfg->thunks_offset;

Cache Line Crossing

The crashing thunk T5's literal is at address 0xf30010cd147c:

Address mod  8 = 4   → NOT naturally aligned for 64-bit atomic access
Address mod 64 = 60  → literal starts at byte 60 of a 64-byte cache line

The 8-byte literal spans two cache lines:

• Cache line 1 (...1440-...147f): contains low 32 bits (0xffbd12a0)
• Cache line 2 (...1480-...14bf): contains high 32 bits (0x0000f2ff)

Per ARM ARM (B2.2.1), a 64-bit access not naturally aligned to 8 bytes is not guaranteed single-copy atomic. When crossing a cache line boundary, the hardware performs two independent operations.

The Torn Read

During runtime thunk patching (emit_thunk at mini-arm64.c:1272), *(guint64*)(code + 8) = (guint64)target writes 0xf2ffffbd12a0 across two cache lines. A concurrent thread executing ldr x16, [pc+8] observes:

Half	Cache Line	Value Seen	Source
Low 32 bits	Line 1	0xffbd12a0	NEW (wrapper)
High 32 bits	Line 2	0x0000f300	OLD (trampoline 0xf3001afe6c90)
Result		0xf300ffbd12a0	Exact crash value ✅

Every 4th thunk in this method has its literal at cache line position 60, making it vulnerable. The bug is rare because it requires concurrent execution during the first-time trampoline resolution.

Proposed Fix

Align the thunk area start to 8 bytes in mini.c:

// Before:
cfg->thunks_offset = cfg->code_size + unwindlen;

// After:
cfg->thunks_offset = ALIGN_TO(cfg->code_size + unwindlen, 8);

The reservation size should also account for the padding:

guint32 thunks_padding = ALIGN_TO(cfg->code_size + unwindlen, 8) - (cfg->code_size + unwindlen);
code = mono_mem_manager_code_reserve(code_mem_manager,
    cfg->code_size + unwindlen + thunks_padding + cfg->thunk_area);

Additionally, a defensive assertion in emit_thunk:

g_assert(((uintptr_t)(code + 8) & 7) == 0); // literal must be 8-byte aligned

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-xml
See info in area-owners.md if you want to be subscribed.