SIGSEGV (signal 11) in RuntimeHelpers.CompileMethod called from LambdaCompiler.CreateDelegate()

Description

After upgrading our service to .NET 10 (10.0.626.17701) and updating several libraries including MongoDB.Driver, the service started crashing regularly with SIGSEGV — 3–4 times per week on the same build. The crashes are non-deterministic: different OS threads each time, same method each time.

The dump shows the crash occurring inside CLR's native RuntimeHelpers.CompileMethod(), invoked from System.Linq.Expressions.Compiler.LambdaCompiler.CreateDelegate() at LambdaCompiler.cs line 265.

The dump also shows two simultaneously live OptimizedTier1 code versions for CreateDelegate — one OptimizedTier1 + Instrumented and one OptimizedTier1 — indicating the JIT was in the middle of a tier transition at crash time.

LambdaExpression.Compile() was being called repeatedly and concurrently from a hot path via MongoDB.Driver LINQ3's PartialEvaluator.SubtreeEvaluator.Evaluate(), which calls Compile() on every invocation of ExpressionFilterDefinition.Render(). CPU had doubled in the 15 minutes before the crash.

Reproduction Steps

The full dump and crashreport.json cannot be shared publicly as they contain references to proprietary application code. The relevant excerpts produced by the analysis script are included in the Script output section below.

The following script reproduces the analysis (requires dotnet-dump, PowerShell 7+):

$DUMP   = "<path to .dmp file>"
$REPORT = "<path to .dmp.crashreport.json file>"

function Main {
    function Section($title) {
        Write-Host "`n$('=' * 70)" -ForegroundColor Cyan
        Write-Host "  $title" -ForegroundColor Cyan
        Write-Host "$('=' * 70)" -ForegroundColor Cyan
    }

    Section "1 — Crashed thread in crashreport.json"
    $hit = Select-String -Path $REPORT -Pattern '"crashed":\s*"true"' | Select-Object -First 1
    Write-Host "  Line $($hit.LineNumber): $($hit.Line.Trim())"
    $done = $false
    Get-Content $REPORT |
        Select-Object -Skip ($hit.LineNumber - 5) -First 220 |
        Select-String -Pattern 'native_thread_id|"is_managed"|"IP"|"SP"|"method_name"' |
        ForEach-Object {
            if ($done) { return }
            $_
            if ($_ -match '"method_name"' -and $_ -match 'FindOneAndUpdateAsync') {
                "      [application code — ...]"
                $done = $true
            }
        }

    Section "2 — clrthreads: crashed thread row (GC mode Cooperative)"
    # Replace 17e1 with native_thread_id from step 1; note DBG column (first) vs managed ID (second)
    "clrthreads`nexit" | dotnet-dump analyze $DUMP 2>&1 | Select-String -Pattern "17e1|^\s+ID\s"

    Section "3 — ip2md: return address into CreateDelegate → method + JIT versions"
    # 0x7f61dcdaacf6 is the return address from CompileMethod back into CreateDelegate @ 265
    "ip2md 0x7f61dcdaacf6`nexit" | dotnet-dump analyze $DUMP 2>&1

    Section "4 — clrstack on crashed thread (use DBG column from step 2, not managed ID)"
    # DBG=923 (first column), managed ID=447 (second column) — these are different
    $done = $false
    "setthread 923`nclrstack`npe`nexit" | dotnet-dump analyze $DUMP 2>&1 | ForEach-Object {
        if ($done) { return }
        $_
        if ($_ -match 'FindOneAndUpdateAsync') {
            "                   [application code — ...]"
            $done = $true
        }
    }

    Section "5 — dumpmd: confirm two live NativeCodeVersions"
    # Replace MethodDesc with value from step 3
    "dumpmd 00007f61d4f956b8`nexit" | dotnet-dump analyze $DUMP 2>&1

    Section "6 — Stack depth (rule out stack overflow)"
    # Extract SP values directly from clrstack output on the crashed thread (use DBG ID from step 2)
    $stackOut = "setthread 923`nclrstack`nexit" | dotnet-dump analyze $DUMP 2>&1
    $spValues = $stackOut | Select-String -Pattern '^\s*(00007F[0-9A-Fa-f]+)\s+[0-9A-Fa-f]+\s' |
        ForEach-Object { [uint64]("0x" + $_.Matches[0].Groups[1].Value) } | Sort-Object
    $min = $spValues | Select-Object -First 1
    $max = $spValues | Select-Object -Last 1
    "  SP range: 0x{0:x} – 0x{1:x}  ({2} bytes = {3:F2} KB used of ~1024 KB limit)" -f $min, $max, ($max - $min), (($max - $min) / 1KB)

    Section "7 — Crash offset: return address into CreateDelegate"
    $codeStart  = [uint64]"0x7f61dcdaab70"   # OptimizedTier1 CodeAddr from step 3
    $returnAddr = [uint64]"0x7f61dcdaacf6"   # return address into CreateDelegate (ip2md input)
    $ctxIP      = [uint64]"0x7f624d8da813"   # ctx.IP from crashreport.json — wait4 in libc.so.6 (signal-handler chain at dump time)
    "  OptimizedTier1 CodeAddr:          0x{0:x}" -f $codeStart
    "  Return addr into CreateDelegate:  0x{0:x}  (+0x{1:x} = {1} bytes)" -f $returnAddr, ($returnAddr - $codeStart)
    "  ctx.IP at dump time:              0x{0:x}  (wait4 in libc.so.6 — signal-handler chain, not the faulting instruction)" -f $ctxIP
    # To disassemble (Linux only):
    #   dotnet-dump analyze <dump> -c "clru 00007f61d4f956b8"
}

Main

Script output:

======================================================================
  1 — Crashed thread in crashreport.json
======================================================================
  Line 174099: "crashed": "true",

    "is_managed": "true",
    "native_thread_id": "0x17e1",
     "IP": "0x7f624d8da813",
     "SP": "0x7f62484691c0",
      "is_managed": "true",
      "is_managed": "true",
      "method_name": "System.Linq.Expressions.LambdaExpression.Compile()",
      "is_managed": "true",
      "method_name": "MongoDB.Driver.Linq.Linq3Implementation.Misc.PartialEvaluator+SubtreeEvaluator.Evaluate(System.Linq.Expressions.Expression)",
      "is_managed": "true",
      "method_name": "MongoDB.Driver.Linq.Linq3Implementation.Misc.PartialEvaluator+SubtreeEvaluator.VisitBinary(System.Linq.Expressions.BinaryExpression)",
      "is_managed": "true",
      "method_name": "MongoDB.Driver.Linq.Linq3Implementation.Misc.PartialEvaluator+SubtreeEvaluator.VisitBinary(System.Linq.Expressions.BinaryExpression)",
      "is_managed": "true",
      "method_name": "MongoDB.Driver.Linq.Linq3Implementation.Misc.PartialEvaluator+SubtreeEvaluator.VisitBinary(System.Linq.Expressions.BinaryExpression)",
      "is_managed": "true",
      "method_name": "System.Linq.Expressions.ExpressionVisitor.VisitLambda[[System.__Canon, System.Private.CoreLib]](System.Linq.Expressions.Expression`1<System.__Canon>)",
      "is_managed": "true",
      "method_name": "MongoDB.Driver.ExpressionFilterDefinition`1[[System.__Canon, System.Private.CoreLib]].Render(MongoDB.Driver.RenderArgs`1<System.__Canon>)",
      "is_managed": "true",
      "method_name": "MongoDB.Driver.MongoCollectionImpl`1[[System.__Canon, System.Private.CoreLib]].CreateFindOneAndUpdateOperation[[System.__Canon, System.Private.CoreLib]](MongoDB.Driver.FilterDefinition`1<System.__Canon>, MongoDB.Driver.UpdateDefinition`1<System.__
Canon>, MongoDB.Driver.FindOneAndUpdateOptions`2<System.__Canon,System.__Canon>)",
      "is_managed": "true",
      "method_name": "MongoDB.Driver.MongoCollectionImpl`1+<FindOneAndUpdateAsync>d__67`1[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].MoveNext()",
      [application code — ...]

======================================================================
  2 — clrthreads: crashed thread row (GC mode Cooperative)
======================================================================
 923  447     17e1 00007F5898088020  1021260 Cooperative 0000000000000000:0000000000000000 000055D8BA71CF00 -00001 Ukn (Threadpool Worker)

======================================================================
  3 — ip2md: return address into CreateDelegate → method + JIT versions
======================================================================
Loading core dump: <path to .dmp file> ...
Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
Type 'quit' or 'exit' to exit the session.
<END_COMMAND_OUTPUT>
> ip2md 0x7f61dcdaacf6
MethodDesc:   00007f61d4f956b8
Method Name:          System.Linq.Expressions.Compiler.LambdaCompiler.CreateDelegate()
Class:                00007f61d4f967b8
MethodTable:          00007f61d4f967b8
mdToken:              0000000006000F6C
Module:               00007f61d05073b8
IsJitted:             yes
Current CodeAddr:     00007f61dcdaab70
Version History:
  ILCodeVersion:      0000000000000000
  ReJIT ID:           0
  IL Addr:            00007f61d0746988
     CodeAddr:           00007f61dc08ba70  (OptimizedTier1 + Instrumented)
     NativeCodeVersion:  00007F5914241AC0
     CodeAddr:           00007f61dcdaab70  (OptimizedTier1)
     NativeCodeVersion:  00007F5914587FE0
     CodeAddr:           00007f61d0598580  (ReadyToRun)
     NativeCodeVersion:  0000000000000000
Source file:  /_/src/runtime/src/libraries/System.Linq.Expressions/src/System/Linq/Expressions/Compiler/LambdaCompiler.cs @ 265
<END_COMMAND_OUTPUT>
> exit

======================================================================
  4 — clrstack on crashed thread (use DBG column from step 2, not managed ID)
======================================================================
Loading core dump: <path to .dmp file> ...
Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
Type 'quit' or 'exit' to exit the session.
<END_COMMAND_OUTPUT>
> setthread 923
<END_COMMAND_OUTPUT>
> clrstack
OS Thread Id: 0x17e1 (923)
        Child SP               IP Call Site
00007F5756D096C0 00007f624d8da813 [InlinedCallFrame: 00007f5756d096c0] System.Runtime.CompilerServices.RuntimeHelpers.CompileMethod(System.RuntimeMethodHandleInternal)
00007F5756D096C0 00007f61dcdaacf6 [InlinedCallFrame: 00007f5756d096c0] System.Runtime.CompilerServices.RuntimeHelpers.CompileMethod(System.RuntimeMethodHandleInternal)
00007F5756D096A0 00007F61DCDAACF6 System.Linq.Expressions.Compiler.LambdaCompiler.CreateDelegate() [/_/src/runtime/src/libraries/System.Linq.Expressions/src/System/Linq/Expressions/Compiler/LambdaCompiler.cs @ 265]
00007F5756D09730 00007F61DCDB19FE System.Linq.Expressions.LambdaExpression.Compile() [/_/src/runtime/src/libraries/System.Linq.Expressions/src/System/Linq/Expressions/LambdaExpression.cs @ 141]
00007F5756D09840 00007F61DF534D27 MongoDB.Driver.Linq.Linq3Implementation.Misc.PartialEvaluator+SubtreeEvaluator.Evaluate(System.Linq.Expressions.Expression)
00007F5756D099C0 00007F61DF534499 MongoDB.Driver.Linq.Linq3Implementation.Misc.PartialEvaluator+SubtreeEvaluator.VisitBinary(System.Linq.Expressions.BinaryExpression)
00007F5756D09A30 00007F61DF53479A MongoDB.Driver.Linq.Linq3Implementation.Misc.PartialEvaluator+SubtreeEvaluator.VisitBinary(System.Linq.Expressions.BinaryExpression)
00007F5756D09AA0 00007F61DF5346CC MongoDB.Driver.Linq.Linq3Implementation.Misc.PartialEvaluator+SubtreeEvaluator.VisitBinary(System.Linq.Expressions.BinaryExpression)
00007F5756D09B10 00007F61DD7C6819 System.Linq.Expressions.ExpressionVisitor.VisitLambda[[System.__Canon, System.Private.CoreLib]](System.Linq.Expressions.Expression`1<System.__Canon>)
00007F5756D09B80 00007F61DF533D92 MongoDB.Driver.ExpressionFilterDefinition`1[[System.__Canon, System.Private.CoreLib]].Render(MongoDB.Driver.RenderArgs`1<System.__Canon>)
00007F5756D09CF0 00007F61E2906053 MongoDB.Driver.MongoCollectionImpl`1[[System.__Canon, System.Private.CoreLib]].CreateFindOneAndUpdateOperation[[System.__Canon, System.Private.CoreLib]](MongoDB.Driver.FilterDefinition`1<System.__Canon>, MongoDB.Driver.UpdateDefinition`1<System.__Canon>, MongoDB.Driver.FindOneAndUpdateOptions`2<System.__Canon,System.__Canon>)
00007F5756D09F90 00007F61E34C0780 MongoDB.Driver.MongoCollectionImpl`1+<FindOneAndUpdateAsync>d__67`1[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]].MoveNext()
                   [application code — ...]
<END_COMMAND_OUTPUT>
> pe
There is no current managed exception on this thread
<END_COMMAND_OUTPUT>
> exit

======================================================================
  5 — dumpmd: confirm two live NativeCodeVersions
======================================================================
Loading core dump: <path to .dmp file> ...
Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
Type 'quit' or 'exit' to exit the session.
<END_COMMAND_OUTPUT>
> dumpmd 00007f61d4f956b8
Method Name:          System.Linq.Expressions.Compiler.LambdaCompiler.CreateDelegate()
Class:                00007f61d4f967b8
MethodTable:          00007f61d4f967b8
mdToken:              0000000006000F6C
Module:               00007f61d05073b8
IsJitted:             yes
Current CodeAddr:     00007f61dcdaab70
Version History:
  ILCodeVersion:      0000000000000000
  ReJIT ID:           0
  IL Addr:            00007f61d0746988
     CodeAddr:           00007f61dc08ba70  (OptimizedTier1 + Instrumented)
     NativeCodeVersion:  00007F5914241AC0
     CodeAddr:           00007f61dcdaab70  (OptimizedTier1)
     NativeCodeVersion:  00007F5914587FE0
     CodeAddr:           00007f61d0598580  (ReadyToRun)
     NativeCodeVersion:  0000000000000000
<END_COMMAND_OUTPUT>
> exit

======================================================================
  6 — Stack depth (rule out stack overflow)
======================================================================
  SP range: 0x7f5756d096a0 – 0x7f5756d0bd30  (9872 bytes = 9,64 KB used of ~1024 KB limit)

======================================================================
  7 — Crash offset: return address into CreateDelegate
======================================================================
  OptimizedTier1 CodeAddr:          0x7f61dcdaab70
  Return addr into CreateDelegate:  0x7f61dcdaacf6  (+0x186 = 390 bytes)
  ctx.IP at dump time:              0x7f624d8da813  (wait4 in libc.so.6 — signal-handler chain, not the faulting instruction)

Expected behavior

LambdaExpression.Compile() called concurrently from multiple ThreadPool threads should not cause
a SIGSEGV. The JIT's tiered recompilation of an internal method should be safe with respect to
concurrent threads executing that method.

Actual behavior

SIGSEGV (signal 11) in the CLR's native RuntimeHelpers.CompileMethod(), called from
LambdaCompiler.CreateDelegate() at LambdaCompiler.cs line 265. The crash is non-deterministic:
three occurrences on the same build, different OS threads each time. The crashed thread's GC mode
is Cooperative (all other threads: Preemptive), confirming it was inside CLR-managed JIT
infrastructure at the time of the fault.

ip2md/dumpmd show two simultaneously live OptimizedTier1 code versions for CreateDelegate
at crash time — OptimizedTier1 + Instrumented (NativeCodeVersion non-null, still live) and
OptimizedTier1 (current) — consistent with the JIT being mid-transition between tiers when the
fault occurred.

Regression?

Unknown. We are running .NET 10 prerelease 10.0.626.17701 and have not tested earlier builds.
The service started crashing after upgrading to this build.

Known Workarounds

No response

Configuration

.NET version: 10.0.626.17701
OS: Linux x64
Container: Docker, Kubernetes pod
Threads at crash time: ~950 OS threads, 298 managed CLR threads

Other information

Hypotheses ruled out:

Hypothesis	Verdict	Evidence
Stack overflow	❌ Ruled out	Stack used ~9.6 KB of ~1 MB ThreadPool limit (SP range: `0x7f5756d096a0`–`0x7f5756d0bd30`)
OOM / memory pressure	❌ Ruled out	Container memory well below limit; pod state `Error` not `OOMKilled`
CPU saturation	❌ Ruled out	~4.7 cores of 8 available at crash time
Lock contention deadlock	❌ Ruled out	Lock contention elevated but produces no SIGSEGV
Native user library crash	❌ Ruled out	Non-managed frames are `libcoreclr.so` + `libc.so.6` (CLR signal handling chain) — no third-party native library
Thrown .NET exception	❌ Ruled out	`pe` on crashed thread: "There is no current managed exception on this thread"

After analyzing additional crash dumps from the same runtime, we see two distinct patterns across ~10 pods.

Pattern 1 — Managed ThreadPool thread: crash at `LambdaCompiler.CreateDelegate()`

In all of these, ip2md shows two simultaneously live NativeCodeVersion entries for LambdaCompiler.CreateDelegate() (one OptimizedTier1+Instrumented, one OptimizedTier1) — consistent with a tiered JIT promotion race.

Example 1 — filter path via ExpressionFilterDefinition + FindOneAndUpdateAsync:

[InlinedCallFrame] RuntimeHelpers.CompileMethod(RuntimeMethodHandleInternal)
LambdaCompiler.CreateDelegate()                         [LambdaCompiler.cs @ 265]
PartialEvaluator+SubtreeEvaluator.Evaluate(Expression)
PartialEvaluator+SubtreeEvaluator.VisitBinary(BinaryExpression)
ExpressionFilterDefinition<T>.Render(RenderArgs<T>)
MongoCollectionImpl<T>.CreateFindOneAndUpdateOperation(...)
MongoCollectionImpl<T>.FindOneAndUpdateAsync(...)
[application code]

Example 2 — filter path via ExpressionFilterDefinition + FindAsync:

[InlinedCallFrame] RuntimeHelpers.CompileMethod(RuntimeMethodHandleInternal)
LambdaCompiler.CreateDelegate()                         [LambdaCompiler.cs @ 265]
PartialEvaluator+SubtreeEvaluator.Evaluate(Expression)
PartialEvaluator+SubtreeEvaluator.VisitBinary(BinaryExpression)   (×2)
ExpressionVisitor.VisitLambda<T>(Expression<T>)
ExpressionFilterDefinition<T>.Render(RenderArgs<T>)
MongoCollectionImpl<T>.CreateFindOperation(...)
MongoCollectionImpl<T>.FindAsync(...)
[application code]

Example 3 — LINQ3 projection path (BsonClassMap.MapConstructor, not a filter):

[InlinedCallFrame] RuntimeHelpers.CompileMethod(RuntimeMethodHandleInternal)
LambdaCompiler.CreateDelegate()                         [LambdaCompiler.cs @ 265]
LambdaCompiler.Compile(LambdaExpression)                [LambdaCompiler.cs @ 192]
BsonClassMap.MapConstructor()
MemberInitExpressionToAggregationExpressionTranslator.Translate()
NewExpressionToAggregationExpressionTranslator.Translate()
ExpressionToAggregationExpressionTranslator.TranslateLambdaBody()
LinqProviderAdapter.TranslateExpressionToProjection()
MongoCollectionImpl<T>.CreateFindOperation(...)
[application code]

Pattern 2 — Native-only thread: crash inside `libcoreclr.so`, no managed frames

In several dumps the crashing thread is absent from clrthreads and has no managed frames at all — the entire stack is libcoreclr.so + libc.so.6. The bottom three offsets (0x557986, 0x495bee, 0x63afb9) are identical across all such dumps and also appear at the bottom of Pattern 1 stacks.

Example 1 — SIGSEGV, 18 frames:

#0   +0x110813  libc.so.6     wait4
#1   +0x638261  libcoreclr.so
#2   +0x6397f3  libcoreclr.so
#3   +0x615e1d  libcoreclr.so
#4   +0x6152f0  libcoreclr.so
#5   +0x045330  libc.so.6
#6   +0x58afd5  libcoreclr.so
#7   +0x58bd74  libcoreclr.so
#8   +0x590b3c  libcoreclr.so
#9   +0x57a9d6  libcoreclr.so
#10  +0x575481  libcoreclr.so
#11  +0x55c1b7  libcoreclr.so
#12  +0x558e64  libcoreclr.so
#13  +0x557986  libcoreclr.so  ← shared fingerprint
#14  +0x495bee  libcoreclr.so  ← shared fingerprint
#15  +0x63afb9  libcoreclr.so  ← shared fingerprint
#16  +0x09caa4  libc.so.6
#17  +0x129c6c  libc.so.6

Example 2 — SIGSEGV, 16 frames (slightly different upper portion):

#0   +0x110813  libc.so.6     wait4
#1   +0x638261  libcoreclr.so
#2   +0x6397f3  libcoreclr.so
#3   +0x615e1d  libcoreclr.so
#4   +0x6152f0  libcoreclr.so
#5   +0x045330  libc.so.6
#6   +0x58c06f  libcoreclr.so
#7   +0x57aa39  libcoreclr.so
#8   +0x575481  libcoreclr.so
#9   +0x55c1b7  libcoreclr.so
#10  +0x558e64  libcoreclr.so
#11  +0x557986  libcoreclr.so  ← shared fingerprint
#12  +0x495bee  libcoreclr.so  ← shared fingerprint
#13  +0x63afb9  libcoreclr.so  ← shared fingerprint
#14  +0x09caa4  libc.so.6
#15  +0x129c6c  libc.so.6

Example 3 — SIGTRAP (signal 5, not SIGSEGV — different signal, same fingerprint):

#0   +0x110813  libc.so.6     wait4
#1   +0x638261  libcoreclr.so
#2   +0x6397f3  libcoreclr.so
#3   +0x636e7d  libcoreclr.so
#4   +0x615139  libcoreclr.so
#5   +0x045330  libc.so.6
#6   +0x2eb405  libcoreclr.so
#7   +0x560f3c  libcoreclr.so
#8   +0x57b65c  libcoreclr.so
#9   +0x575481  libcoreclr.so
#10  +0x55c1b7  libcoreclr.so
#11  +0x558e64  libcoreclr.so
#12  +0x557986  libcoreclr.so  ← shared fingerprint
#13  +0x495bee  libcoreclr.so  ← shared fingerprint
#14  +0x63afb9  libcoreclr.so  ← shared fingerprint
#15  +0x09caa4  libc.so.6
#16  +0x129c6c  libc.so.6

The three offsets (0x557986, 0x495bee, 0x63afb9) appear in every crash dump we have — both patterns — which strongly suggests they share the same root cause and only differ in which thread is executing the racing code at the moment it corrupts memory.

@flibustier7seas

@flibustier7seas just to symbolicate the crash, is it possible to run

dotnet-symbol --symbols libcoreclr.so
dotnet-symbol --symbols libclrjit.so

the tool can be installed with dotnet tool install --global dotnet-symbol (https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-symbol)

and is it possible to share the dump?

@flibustier7seas

@flibustier7seas just to symbolicate the crash, is it possible to run

@EgorBo I tried to symbolicate the crash dump. Here is what I got:

root@81d781a88e31:/# apt-get update -qq && apt-get install -y -qq wget

root@81d781a88e31:/# wget -q https://dot.net/v1/dotnet-install.sh

root@81d781a88e31:/# chmod +x dotnet-install.sh

root@81d781a88e31:/# ./dotnet-install.sh --channel 10.0

...

root@81d781a88e31:/# export PATH="$HOME/.dotnet:$PATH"

root@81d781a88e31:/# dotnet tool install --global dotnet-symbol

...

root@81d781a88e31:/# export PATH="$PATH:$HOME/.dotnet/tools"

root@81d781a88e31:/# cd /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.6/

root@81d781a88e31:/usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.6# dotnet-symbol --symbols libcoreclr.so

Downloading from https://msdl.microsoft.com/download/symbols/
Writing: ./libcoreclr.so.dbg

root@81d781a88e31:/usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.6# dotnet-symbol --symbols libclrjit.so

Downloading from https://msdl.microsoft.com/download/symbols/
Writing: ./libclrjit.so.dbg

root@81d781a88e31:/usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.6# apt-get install -y -qq lldb

...

root@81d781a88e31:/usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.6# lldb --core /dump/crash.dmp \
     -o "settings set target.debug-file-search-paths /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.6" \
     -o "thread select 7" \
     -o "bt" \
     -o "quit"

(lldb) target create --core "/dump/crash.dmp"
Core file '/dump/crash.dmp' (x86_64) was loaded.
(lldb) settings set target.debug-file-search-paths /usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.6
(lldb) thread select 7
(lldb) bt
* thread #7, stop reason = signal SIGTRAP
    frame #0: 0x00007f2d8374e813 libc.so.6`wait4 + 83
libc.so.6`wait4:
->  0x7f2d8374e813 <+83>: cmpq   $-0x1000, %rax ; imm = 0xF000
    0x7f2d8374e819 <+89>: ja     0x7f2d8374e848 ; <+136>
    0x7f2d8374e81b <+91>: movl   %r8d, %edi
    0x7f2d8374e81e <+94>: movl   %eax, -0x4(%rbp)
* thread #7, stop reason = signal SIGTRAP
  * frame #0: 0x00007f2d8374e813 libc.so.6`wait4 + 83
    frame #1: 0x00007f2d834b0261 libcoreclr.so`PROCCreateCrashDump(argv=size=17, errorMessageBuffer=0x0000000000000000, cbErrorMessageBuffer=0, serialize=<unavailable>) at process.cpp:2580:22 [opt]
    frame #2: 0x00007f2d834b17f3 libcoreclr.so`PROCCreateCrashDumpIfEnabled(signal=<unavailable>, siginfo=<unavailable>, serialize=true) at process.cpp:2807:9 [opt]
    frame #3: 0x00007f2d834aee7d libcoreclr.so`PROCAbort(signal=5, siginfo=0x00007f2d803f8ef0) at process.cpp:2839:5 [opt]
    frame #4: 0x00007f2d8348d139 libcoreclr.so`sigtrap_handler(int, siginfo_t*, void*) [inlined] invoke_previous_action(action=<unavailable>, code=5, siginfo=0x00007f2d803f8ef0, context=0x00007f2d803f8dc0, signalRestarts=false) at signal.cpp:447:13 [opt]
    frame #5: 0x00007f2d8348d11f libcoreclr.so`sigtrap_handler(code=5, siginfo=0x00007f2d803f8ef0, context=0x00007f2d803f8dc0) at signal.cpp:753:5 [opt]
    frame #6: 0x00007f2d83683330 libc.so.6`___lldb_unnamed_symbol3390 + 1
    frame #7: 0x00007f2d83163405 libcoreclr.so`GCToOSInterface::DebugBreak() at gcenv.unix.cpp:507:1 [opt]
    frame #8: 0x00007f2d833d8f3c libcoreclr.so`SVR::gc_heap::sort_mark_list() [inlined] SVR::FATAL_GC_ERROR() at gcpriv.h:116:5 [opt]
    frame #9: 0x00007f2d833d8f37 libcoreclr.so`SVR::gc_heap::sort_mark_list(this=0x0000562a3c67fc00) at gc.cpp:10798:13 [opt]
    frame #10: 0x00007f2d833f365c libcoreclr.so`SVR::gc_heap::mark_phase(this=0x0000562a3c67fc00, condemned_gen_number=0) at gc.cpp:30601:35 [opt]
    frame #11: 0x00007f2d833ed481 libcoreclr.so`SVR::gc_heap::gc1(this=0x0000562a3c67fc00) at gc.cpp:22707:13 [opt]
    frame #12: 0x00007f2d833d41b7 libcoreclr.so`SVR::gc_heap::garbage_collect(this=0x0000562a3c67fc00, n=<unavailable>) at gc.cpp:0 [opt]
    frame #13: 0x00007f2d833d0e64 libcoreclr.so`SVR::gc_heap::gc_thread_function(this=0x0000562a3c67fc00) at gc.cpp:7312:13 [opt]
    frame #14: 0x00007f2d833cf986 libcoreclr.so`SVR::gc_heap::gc_thread_stub(arg=0x0000562a3c67fc00) at gc.cpp:38021:11 [opt]
    frame #15: 0x00007f2d8330dbee libcoreclr.so`(anonymous namespace)::CreateNonSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::__invoke(void*) [inlined] (anonymous namespace)::CreateNonSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::operator()(this=<unavailable>, argument=<unavailable>) const at gcenv.ee.cpp:1595:13 [opt]
    frame #16: 0x00007f2d8330dbad libcoreclr.so`(anonymous namespace)::CreateNonSuspendableThread(void (*)(void*), void*, char16_t const*)::$_0::__invoke(argument=<unavailable>) at gcenv.ee.cpp:1576:27 [opt]
    frame #17: 0x00007f2d834b2fb9 libcoreclr.so`CorUnix::CPalThread::ThreadEntry(pvParam=0x0000562a3c67e630) at thread.cpp:1622:16 [opt]
    frame #18: 0x00007f2d836daaa4 libc.so.6`___lldb_unnamed_symbol3670 + 900
    frame #19: 0x00007f2d83767c6c libc.so.6`___lldb_unnamed_symbol4062 + 7
(lldb) quit

root@81d781a88e31:/usr/share/dotnet/shared/Microsoft.NETCore.App/10.0.6#

and is it possible to share the dump?

I will check internally whether we can share the dump file.

Thanks, given the crash happens in the GC, I'm assigning to the @dotnet/gc, perhaps, the stack trace looks familiar

The error we are hitting is this FATAL_GC_ERROR() here:
https://github.com/dotnet/runtime/blob/7706f546bac1a99b3d891afe3591dc88c67f0cc4/src/coreclr/gc/gc.cpp#L10793-L10799

The comment on this says:

// Due to GC holes, x can point to something in a region that already got freed. And that region's
// allocated would be 0 and cause an infinite loop which is much harder to handle on production than
// simply throwing an exception.

This seems to indicate that the primary cause is likely a GC hole. To figure out the problem, a standalone repro would be ideal. We might also find more details from a dump. Without any of these, it is hard to say where is the origin of this problem.

To figure out the problem, a standalone repro would be ideal.

I don't have a standalone reproduction yet - the crashes occur on a large production service running many different workloads simultaneously.

However, in roughly half of the crash dumps the managed thread stacks show MongoDB driver query building via ExpressionFilterDefinition → PartialEvaluator → LambdaExpression.Compile(). I'll try to build a minimal repro around that pattern.

We might also find more details from a dump.

I can't share the full file at the moment, but I'm happy to extract and provide specific data from it.

@flibustier7seas

@flibustier7seas were you able to make a minimal repro around it? Any progress in

@kkokosa

@kkokosa No, unfortunately not - I haven't managed to reproduce it in a standalone project.

I also can't share the memory dumps from the real application, but if there's something specific that would help, I can try to pull out particular fragments from them - just let me know what to look for.

@kkokosa

@kkokosa @janvorli Hi!

Reproduced: https://github.com/flibustier7seas/dotnet-sigsegv-repro

docker compose up --build on 10.0.3-noble, 4 vCPU / 2.5 GB container limit, server GC with DATAS.

Time to crash ranges from 5 minutes to 5 hours.

The fault matches the managed-thread half of our production crashes: SIGSEGV in LCGMethodResolver::GetCodeInfo (dynamicmethod.cpp:1190), reading the object header of the byte[] the managed resolver just returned.

I have the dump (~5 GB) if you want it.

Can it be related to #131267 ?

It is fixed in main (is part of .NET 11) and also backported to .NET 10, merged 2026-08-03, targeting milestone 10.0.12 (not yet shipped). So yes, this is the same issue as #131267 - that one was closed as "Fixed by #131708", the same backport.

Your repro vs various versions:

Runtime	Fix	Crashes / runs
10.0.3 (your repro default)	no	1 / 1
10.0.6 (your production version)	no	1 / 15
10.0.11 (latest patch)	no	5 / 12
`release/10.0` @ `f8a7519` (10.0.12 candidate)	yes	0 / 14
11.0.0-rc.1	yes	0 / 7

Navigation Menu

Repository navigation

SIGSEGV (signal 11) in RuntimeHelpers.CompileMethod called from LambdaCompiler.CreateDelegate() #128255

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

Pattern 1 — Managed ThreadPool thread: crash at `LambdaCompiler.CreateDelegate()`

Pattern 2 — Native-only thread: crash inside `libcoreclr.so`, no managed frames

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SIGSEGV (signal 11) in RuntimeHelpers.CompileMethod called from LambdaCompiler.CreateDelegate() #128255

Description

Description

Reproduction Steps

Expected behavior

Actual behavior

Regression?

Known Workarounds

Configuration

Other information

Activity

flibustier7seas commented on May 15, 2026

Pattern 1 — Managed ThreadPool thread: crash at LambdaCompiler.CreateDelegate()

Pattern 2 — Native-only thread: crash inside libcoreclr.so, no managed frames

EgorBo commented on May 15, 2026

flibustier7seas commented on May 16, 2026

EgorBo commented on May 16, 2026

janvorli commented on May 18, 2026

flibustier7seas commented on May 18, 2026

kkokosa commented on Aug 10, 2026

flibustier7seas commented on Aug 14, 2026

flibustier7seas commented on Aug 21, 2026

flibustier7seas commented on Aug 23, 2026

kkokosa commented on Aug 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Pattern 1 — Managed ThreadPool thread: crash at `LambdaCompiler.CreateDelegate()`

Pattern 2 — Native-only thread: crash inside `libcoreclr.so`, no managed frames