[linux] Segfault while invoking Roslyn's csc.dll with long command line

[grendello]

The crash happens occasionally (every 4-5 builds) when building .NET Android's Mono.Android project, which passes a very long command line to the Roslyn C# compiler (1.8Mb long, over 11k parameters). It started happening with bump to the latest dotnet 9 preview, dotnet/installer/main@0a73f814e1 9.0.100-preview.2.24122.3.

Environment info:

OS: Debian/testing (trixie)
Arch: amd64
CPU: AMD Ryzen 9 5950X 16-Core Processor
Kernel: 6.7.6-x64v3-xanmod1

The crash doesn't happen on every build, but it appears the more parallel the build, the crashier the process becomes. It also appears that putting the long command line in a response file mitigates the issue, at least I wasn't able to reproduce the crash this way after 20 attempts. When the crash happens, runtime just goes away silently without capturing the signal and printing any information. Our CI Linux bot uses Ubuntu 20.04.6 LTS and, so far, doesn't appear to experience this crash (or we were lucky, as it only landed last Friday)

Unfortunately, to reproduce locally one would have to build Xamarin.Android on their machine and extract the csc invocation from binlog. This is necessary because the build uses a lot of artifacts produced by .NET Android build. I will be happy to provide assistance in that regard, if necessary :)

I extracted the compiler invocation into a shell script and captured a handful of coredumps, the best trace I managed to get is below:

* thread #1, name = 'dotnet', stop reason = signal SIGSEGV: address not mapped to object
  * frame #0: 0x00007f1f37e58384 libcoreclr.so`MethodTable::CheckRestore() [inlined] MethodTable::GetAuxiliaryData(this=0x0000000000000000) const at methodtable.h:2690:29 [opt]
    frame #1: 0x00007f1f37e58384 libcoreclr.so`MethodTable::CheckRestore() [inlined] MethodTable::IsFullyLoaded(this=0x0000000000000000) at methodtable.h:943:17 [opt]
    frame #2: 0x00007f1f37e58384 libcoreclr.so`MethodTable::CheckRestore(this=0x0000000000000000) at methodtable.cpp:4828:10 [opt]
    frame #3: 0x00007f1f37f324ef libcoreclr.so`JIT_Box(type=0x0000000000000000, unboxedData=<unavailable>) at jithelpers.cpp:2952:10 [opt]
    frame #4: 0x00007f1ec095f5b8
    frame #5: 0x00007f1eb8dceb12
    frame #6: 0x00007f1ebf7edde9
    frame #7: 0x00007f1ebe3c7879
    frame #8: 0x00007f1ebf7cfb66
    frame #9: 0x00007f1ebf7e15f6
    frame #10: 0x00007f1ebf7cb8f8
    frame #11: 0x00007f1ebf7caea2
    frame #12: 0x00007f1ebf7c9300
    frame #13: 0x00007f1ebf7c81e3
    frame #14: 0x00007f1ebf7c810e
    frame #15: 0x00007f1ebf7ca3b3
    frame #16: 0x00007f1ebf7c8db1
    frame #17: 0x00007f1ebf7c81e3
    frame #18: 0x00007f1ebf7c810e
    frame #19: 0x00007f1ebf7c7fbf
    frame #20: 0x00007f1ebf7c7921
    frame #21: 0x00007f1ebf7c2fce
    frame #22: 0x00007f1ebf7c0ef6
    frame #23: 0x00007f1ebd3558c3
    frame #24: 0x00007f1ebd354589
    frame #25: 0x00007f1ec09ce820
    frame #26: 0x00007f1ec09ce421
    frame #27: 0x00007f1ec099e1bb
    frame #28: 0x00007f1ec09a9fa4
    frame #29: 0x00007f1ec09a87f6
    frame #30: 0x00007f1ebf1fcb69
    frame #31: 0x00007f1ebf1de387
    frame #32: 0x00007f1ec09aedd6
    frame #33: 0x00007f1ec09a7e45
    frame #34: 0x00007f1ec09c1066
    frame #35: 0x00007f1ec09c5702
    frame #36: 0x00007f1ec09c0742
    frame #37: 0x00007f1ec09c04f2
    frame #38: 0x00007f1ec09c0078
    frame #39: 0x00007f1ec09bfc8e
    frame #40: 0x00007f1ec09bfb56
    frame #41: 0x00007f1ec09bd975
    frame #42: 0x00007f1eb9e856dd
    frame #43: 0x00007f1ebe4b8970
    frame #44: 0x00007f1ebe4b8640
    frame #45: 0x00007f1eb9e00752
    frame #46: 0x00007f1eb9e9e820
    frame #47: 0x00007f1ebe4b84d8
    frame #48: 0x00007f1ebe4b8450
    frame #49: 0x00007f1eb9e82458
    frame #50: 0x00007f1eb912d821
    frame #51: 0x00007f1ebe023091
    frame #52: 0x00007f1ebe020a24
    frame #53: 0x00007f1ebe46b422
    frame #54: 0x00007f1eb8eae5ab
    frame #55: 0x00007f1f380989a7 libcoreclr.so`CallDescrWorkerInternal at calldescrworkeramd64.S:97
    frame #56: 0x00007f1f37ed2cb6 libcoreclr.so`DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) [inlined] CallDescrWorkerWithHandler(pCallDescrData=0x00007edde5bffbc8, fCriticalCall=NO) at callhelpers.cpp:67:5 [opt]
    frame #57: 0x00007f1f37ed2c5e libcoreclr.so`DispatchCallSimple(pSrc=<unavailable>, numStackSlotsToCopy=<unavailable>, pTargetAddress=<unavailable>, dwDispatchCallSimpleFlags=0) at callhelpers.cpp:218:9 [opt]
    frame #58: 0x00007f1f37ee8692 libcoreclr.so`ThreadNative::KickOffThread_Worker(ptr=<unavailable>) at comsynchronizable.cpp:157:5 [opt]
    frame #59: 0x00007f1f37ea1a45 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchInner(pCallState=<unavailable>) at threads.cpp:7276:5 [opt]
    frame #60: 0x00007f1f37ea1a43 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchMiddle(pCallState=<unavailable>) at threads.cpp:7320:9 [opt]
    frame #61: 0x00007f1f37ea1a08 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchOuter(this=<unavailable>, pParam=<unavailable>)::$_0::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const::'lambda'(Param*)::operator()(Param*) const at threads.cpp:7478:13 [opt]
    frame #62: 0x00007f1f37ea1a08 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_0::operator()(this=<unavailable>, pArgs=<unavailable>) const at threads.cpp:7480:9 [opt]
    frame #63: 0x00007f1f37ea19e5 libcoreclr.so`ManagedThreadBase_DispatchOuter(pCallState=0x00007edde5bffda0) at threads.cpp:7504:5 [opt]
    frame #64: 0x00007f1f37ea200d libcoreclr.so`ManagedThreadBase::KickOff(void (*)(void*), void*) [inlined] ManagedThreadBase_FullTransition(pTarget=<unavailable>, args=<unavailable>, filterType=ManagedThread) at threads.cpp:7524:5 [opt]
    frame #65: 0x00007f1f37ea1ff5 libcoreclr.so`ManagedThreadBase::KickOff(pTarget=<unavailable>, args=<unavailable>) at threads.cpp:7559:5 [opt]
    frame #66: 0x00007f1f37ee8768 libcoreclr.so`ThreadNative::KickOffThread(pass=0x00007edde8004080) at comsynchronizable.cpp:228:9 [opt]
    frame #67: 0x00007f1f3820916e libcoreclr.so`CorUnix::CPalThread::ThreadEntry(pvParam=0x00007edde8001cf0) at thread.cpp:1760:16 [opt]
    frame #68: 0x00007f1f383c745c libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:444:8
    frame #69: 0x00007f1f38447bbc libc.so.6`__clone3 at clone3.S:81

[MichalPetryka]

AFAIR Linux doesn't support such line lenghts and it truncates them, can't find a reliable answer on google though since various results say that either 1024, 2048, 4096 or 128k chars is the limit.

[grendello]

The line limits are higher than on macOS (which has 1M):

$ getconf ARG_MAX
2097152

And it appears that OS-imposed line length limit is not a factor here, since named pipes appear to be used to pass arguments to the sub-process. Command line length may play a role in either dotnet or Roslyn argument parsing, but it might be a red herring too.

[grendello]

Just tested with the new dotnet, 9.0.0-preview.2.24123.1, and the crash still happens in the same way

[mangod9]

Hello @grendello, are you able to share the dump and/or a repro for this issue?

[grendello]

Hello @mangod9, I didn't save any of the original core dump files, alas, but I've just tested with dotnet 9.0.100-preview.5.24229.2 and wasn't able to reproduce this issue anymore. It seems that it "fixed itself" :)

I will close the issue and reopen it only should I see the crash again. Thanks!

[grendello]

@mangod9 well, it started happening again with the update to 9.0.100-preview.5.24262.2 and here's the trace for the crash:

* thread #1, name = 'dotnet', stop reason = signal SIGSEGV: address not mapped to object
  * frame #0: 0x00007fdc72474cae libcoreclr.so`WKS::gc_heap::mark_object_simple(unsigned char**) [inlined] MethodTable::GetFlag(this=0x0000000000000000, flag=enum_flag_HasComponentSize) const at methodtable.h:3476:16 [opt]
    frame #1: 0x00007fdc72474cae libcoreclr.so`WKS::gc_heap::mark_object_simple(unsigned char**) [inlined] MethodTable::HasComponentSize(this=0x0000000000000000) const at methodtable.h:1497:16 [opt]
    frame #2: 0x00007fdc72474cae libcoreclr.so`WKS::gc_heap::mark_object_simple(unsigned char**) [inlined] WKS::my_get_size(ob=0x00007f9c3982df08) at gc.cpp:11491:18 [opt]
    frame #3: 0x00007fdc72474ca7 libcoreclr.so`WKS::gc_heap::mark_object_simple(po=0x00007f9bd7fe12d8) at gc.cpp:27782:24 [opt]
    frame #4: 0x00007fdc7247be1a libcoreclr.so`WKS::gc_heap::mark_through_cards_for_uoh_objects(void (*)(unsigned char**), int, int) [inlined] WKS::gc_heap::mark_through_cards_helper(poo=0x00007f9bd7fe12d8, n_gen=<unavailable>, cg_pointers_found=0x00007fdbe8bfd3b8, fn=(libcoreclr.so`WKS::gc_heap::mark_object_simple(unsigned char**) at gc.cpp:27761), nhigh=0x0000000000000000, next_boundary=0x0000000000000000, condemned_gen=0, current_gen=2) at gc.cpp:41242:9 [opt]
    frame #5: 0x00007fdc7247bda8 libcoreclr.so`WKS::gc_heap::mark_through_cards_for_uoh_objects(fn=(libcoreclr.so`WKS::gc_heap::mark_object_simple(unsigned char**) at gc.cpp:27761), gen_num=3, relocating=NO) at gc.cpp:46720:21 [opt]
    frame #6: 0x00007fdc72469321 libcoreclr.so`WKS::gc_heap::mark_phase(condemned_gen_number=0) at gc.cpp:29764:25 [opt]
    frame #7: 0x00007fdc72464ddc libcoreclr.so`WKS::gc_heap::gc1() at gc.cpp:22184:13 [opt]
    frame #8: 0x00007fdc724726ab libcoreclr.so`WKS::gc_heap::garbage_collect(n=0) at gc.cpp:0:5 [opt]
    frame #9: 0x00007fdc7245ff89 libcoreclr.so`WKS::GCHeap::GarbageCollectGeneration(this=<unavailable>, gen=0, reason=reason_alloc_soh) at gc.cpp:50736:9 [opt]
    frame #10: 0x00007fdc7246156a libcoreclr.so`WKS::gc_heap::trigger_gc_for_alloc(gen_number=<unavailable>, gr=<unavailable>, msl=0x00007fdc726bbda8, loh_p=<unavailable>, take_state=<unavailable>) at gc.cpp:18766:14 [opt]
    frame #11: 0x00007fdc724621e0 libcoreclr.so`WKS::gc_heap::try_allocate_more_space(acontext=0x00007f9b0c001aa0, size=40, flags=2, gen_number=0) at gc.cpp:18904:34 [opt]
    frame #12: 0x00007fdc72491f90 libcoreclr.so`WKS::GCHeap::Alloc(gc_alloc_context*, unsigned long, unsigned int) [inlined] WKS::gc_heap::allocate_more_space(acontext=0x00007f9b0c001aa0, size=40, flags=2, alloc_generation_number=0) at gc.cpp:19404:18 [opt]
    frame #13: 0x00007fdc72491f75 libcoreclr.so`WKS::GCHeap::Alloc(gc_alloc_context*, unsigned long, unsigned int) [inlined] WKS::gc_heap::allocate(jsize=40, acontext=0x00007f9b0c001aa0, flags=2) at gc.cpp:19435:19 [opt]
    frame #14: 0x00007fdc72491f5a libcoreclr.so`WKS::GCHeap::Alloc(this=<unavailable>, context=0x00007f9b0c001aa0, size=40, flags=2) at gc.cpp:49660:34 [opt]
    frame #15: 0x00007fdc72317c7e libcoreclr.so`Alloc(size=40, flags=GC_ALLOC_CONTAINS_REF) at gchelpers.cpp:227:48 [opt]
    frame #16: 0x00007fdc72318e01 libcoreclr.so`AllocateObject(pMT=0x00007fdbf9090840, flags=GC_ALLOC_CONTAINS_REF) at gchelpers.cpp:1095:37 [opt]
    frame #17: 0x00007fdc7233441f libcoreclr.so`JIT_New(CORINFO_CLASS_STRUCT_*) [inlined] AllocateObject(pMT=0x00007fdbf9090840) at gchelpers.h:68:12 [opt]
    frame #18: 0x00007fdc72334418 libcoreclr.so`JIT_New(typeHnd_=0x00007fdbf9090840) at jithelpers.cpp:2353:14 [opt]
    frame #19: 0x00007fdbf8882098
    frame #20: 0x00007fdbf8810e09
    frame #21: 0x00007fdbfab4de09
    frame #22: 0x00007fdbf979db9d
    frame #23: 0x00007fdbf96ef4b7
    frame #24: 0x00007fdbf516c8fd
    frame #25: 0x00007fdbf51825d1
    frame #26: 0x00007fdbf76f3122
    frame #27: 0x00007fdbf76f1348
    frame #28: 0x00007fdbf76ecfa7
    frame #29: 0x00007fdbf9716a33
    frame #30: 0x00007fdbf97166c8
    frame #31: 0x00007fdbf41fa64d
    frame #32: 0x00007fdbf41f6317
    frame #33: 0x00007fdbf427b984
    frame #34: 0x00007fdbf9710212
    frame #35: 0x00007fdbf9710180
    frame #36: 0x00007fdbf41f613f
    frame #37: 0x00007fdbf427b587
    frame #38: 0x00007fdbf97100c2
    frame #39: 0x00007fdbf9710030
    frame #40: 0x00007fdbf41f607f
    frame #41: 0x00007fdbf427b2a9
    frame #42: 0x00007fdbf970ff76
    frame #43: 0x00007fdbf970fedc
    frame #44: 0x00007fdbf41f5f22
    frame #45: 0x00007fdbf427b064
    frame #46: 0x00007fdbf970fd10
    frame #47: 0x00007fdbf970f9b0
    frame #48: 0x00007fdbf41f5e92
    frame #49: 0x00007fdbf42942c4
    frame #50: 0x00007fdbf970f848
    frame #51: 0x00007fdbf970f7c0
    frame #52: 0x00007fdbf4277db8
    frame #53: 0x00007fdbf3522371
    frame #54: 0x00007fdbf88b1ab5
    frame #55: 0x00007fdbf88afeaa
    frame #56: 0x00007fdbf88aede0
    frame #57: 0x00007fdbf32aa54f
    frame #58: 0x00007fdc7249a2ff libcoreclr.so`CallDescrWorkerInternal at calldescrworkeramd64.S:97
    frame #59: 0x00007fdc722d8155 libcoreclr.so`DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) [inlined] CallDescrWorkerWithHandler(pCallDescrData=0x00007fdbe8bffbb8, fCriticalCall=NO) at callhelpers.cpp:67:5 [opt]
    frame #60: 0x00007fdc722d80fe libcoreclr.so`DispatchCallSimple(pSrc=<unavailable>, numStackSlotsToCopy=<unavailable>, pTargetAddress=<unavailable>, dwDispatchCallSimpleFlags=0) at callhelpers.cpp:218:9 [opt]
    frame #61: 0x00007fdc722ed472 libcoreclr.so`ThreadNative::KickOffThread_Worker(ptr=<unavailable>) at comsynchronizable.cpp:157:5 [opt]
    frame #62: 0x00007fdc722a7295 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchInner(pCallState=<unavailable>) at threads.cpp:7259:5 [opt]
    frame #63: 0x00007fdc722a7293 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchMiddle(pCallState=<unavailable>) at threads.cpp:7303:9 [opt]
    frame #64: 0x00007fdc722a7247 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchOuter(this=<unavailable>, pParam=<unavailable>)::$_0::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const::'lambda'(Param*)::operator()(Param*) const at threads.cpp:7461:13 [opt]
    frame #65: 0x00007fdc722a7247 libcoreclr.so`ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) [inlined] ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_0::operator()(this=<unavailable>, pArgs=<unavailable>) const at threads.cpp:7463:9 [opt]
    frame #66: 0x00007fdc722a7221 libcoreclr.so`ManagedThreadBase_DispatchOuter(pCallState=0x00007fdbe8bffda0) at threads.cpp:7487:5 [opt]
    frame #67: 0x00007fdc722a766d libcoreclr.so`ManagedThreadBase::KickOff(void (*)(void*), void*) [inlined] ManagedThreadBase_FullTransition(pTarget=<unavailable>, args=<unavailable>, filterType=ManagedThread) at threads.cpp:7507:5 [opt]
    frame #68: 0x00007fdc722a7655 libcoreclr.so`ManagedThreadBase::KickOff(pTarget=<unavailable>, args=<unavailable>) at threads.cpp:7542:5 [opt]
    frame #69: 0x00007fdc722ed548 libcoreclr.so`ThreadNative::KickOffThread(pass=0x00007f9b0c001a50) at comsynchronizable.cpp:228:9 [opt]
    frame #70: 0x00007fdc72624fce libcoreclr.so`CorUnix::CPalThread::ThreadEntry(pvParam=0x00007f9b0c002fb0) at thread.cpp:1760:16 [opt]
    frame #71: 0x00007fdc729c2dbb libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:444:8
    frame #72: 0x00007fdc72a449f8 libc.so.6`__clone3 at clone3.S:78

[grendello]

In hope to get better traces, I built dotnet/runtime's main branch (at commit 9e57de2` in Debug configuration but, alas, the segfault doesn't happen.
What happens in this instance, however, is that where the runtime would segfault it hits the following assertion:

dotnet/sdk/9.0.100-preview.6.24319.5/Roslyn/Microsoft.CSharp.Core.targets(85,5): error : Assert failure(PID 74970 [0x000124da], Thread: 75151 [0x1258f]): Consistency check failed: Crst Level violation: Can't take level 17 lock CrstLoaderAllocator because you already holding level 5 lock CrstThreadLocalStorageLock
dotnet/sdk/9.0.100-preview.6.24319.5/Roslyn/Microsoft.CSharp.Core.targets(85,5): error : FAILED: false
dotnet/sdk/9.0.100-preview.6.24319.5/Roslyn/Microsoft.CSharp.Core.targets(85,5): error :     File: runtime/src/coreclr/vm/crst.cpp:765

[grendello]

Good news is that dotnet/runtime main branch built with build.sh -c release -gcc-14 doesn't crash, however without knowing the codebase it's hard for me to say whether the problem is fixed or (as it previously happened) whether it "vanished" for a while.