[7.0][mono][arm64] futex-internal.h: No such file or directory - futex_wait_cancelable native crash in Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests

[carlossanlop]

Error Blob

Unsure if this is a known issue or not. I didn't find an open with this method failing in the callstack. Feel free to close as dup if you find one.

{
  "ErrorMessage": "futex-internal.h: No such file or directory",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

• Affected 7.0 PR: [release/7.0-staging] Reset OOB packages from 7.0.7 #86300
• Affected test: Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests
• Job: Libraries Test Run release mono Linux arm64 Debug
• Job link: https://dev.azure.com/dnceng-public/public/_build/results?buildId=304452&view=logs&j=3f95ca23-483b-5dc9-4b05-6bd24fb925c2&t=0c1dd9d1-4ef7-54d5-2d62-e4ab330605ca&l=56
• Queue: (Ubuntu.1804.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
• Full callstack:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0xffaf543edff0 (LWP 25) "dotnet" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0xffaf533ff1c0 (LWP 26) "SGen worker" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0xffaf516921c0 (LWP 27) "dotnet" 0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  4    Thread 0xffaf514911c0 (LWP 28) "Finalizer" 0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0xffaf491de1c0 (LWP 29) "dotnet" 0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
  6    Thread 0xffaf490c71c0 (LWP 32) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    Thread 0xffaf48ec61c0 (LWP 33) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  8    Thread 0xffaf48b3b1c0 (LWP 34) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  9    Thread 0xffaf48ada1c0 (LWP 35) ".NET ThreadPool" mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
  10   Thread 0xffaf488861c0 (LWP 36) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  11   Thread 0xffaf486851c0 (LWP 37) ".NET Long Runni" 0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
  12   Thread 0xffaf276891c0 (LWP 39) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142

Thread 12 (Thread 0xffaf276891c0 (LWP 39)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf27688340, mutex=0xffaf380c8520, cond=0xffaf27688390) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, abstime=0xffaf27688340) at pthread_cond_wait.c:667
#3  0x0000ffaf53c45e98 in mono_os_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#4  0x0000ffaf53c4b494 in mono_coop_cond_timedwait (cond=0xffaf27688390, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#5  mono_lifo_semaphore_timed_wait (semaphore=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#6  0x0000ffaf48aea098 in ?? ()
#7  0x0000ffaf53ba6f28 in ves_icall_System_Threading_Thread_YieldInternal () at /__w/1/s/src/mono/mono/metadata/icall.c:7185
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 11 (Thread 0xffaf486851c0 (LWP 37)):
#0  0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ffaf53d8706c in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xffaf4867ead0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ffaf53d455ac in mono_handle_native_crash (signal=0xffaf53b47770 "SIGSEGV", mctx=0xffaf4867ead0, info=0xffaf4867ee30) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ffaf53cafa80 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xffaf4867ee30, context=0xffaf4867eeb0, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3774
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ffaf2789be98 in ?? ()
#8  0x0000ffaf5347b3a8 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 10 (Thread 0xffaf488861c0 (LWP 36)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf2800e960, cond=0xffaf2800e990) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf2800e990, mutex=0xffaf2800e960) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf2800e960) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000000000000002 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 9 (Thread 0xffaf48ada1c0 (LWP 35)):
#0  mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
#1  0x0000ffaf53ca673c in mono_codegen (cfg=0xffaf357a7990) at /__w/1/s/src/mono/mono/mini/mini.c:2212
#2  0x0000ffaf53ca8da0 in mini_method_compile (method=<optimized out>, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3935
#3  0x0000ffaf53caa0b8 in mono_jit_compile_method_inner (method=0xffaf34f93480, opt=172, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini.c:4129
#4  0x0000ffaf53caeba0 in mono_jit_compile_method_with_opt (method=0xffaf34f93480, opt=374417919, jit_only=0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2708
#5  jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2763
#6  jit_compile_method_with_opt (params=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2779
#7  0x0000ffaf53cae018 in mono_jit_compile_method (method=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2798
#8  0x0000ffaf53d47a18 in common_call_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", m=0xffaf34f93480, vt=0xffaf3540ef10, vtable_slot=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618
#9  0x0000ffaf53d48788 in mono_vcall_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", slot=1, tramp=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:840
#10 0x0000ffaf53920504 in ?? ()
#11 0x0000ffaf5368b8d0 in ?? ()
#12 0x0000ffaf5368cec8 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind further

Thread 8 (Thread 0xffaf48b3b1c0 (LWP 34)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48b3a2a8, mutex=0xffaf30020ed0, cond=0xffaf30020f00) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf30020f00, mutex=0xffaf30020ed0, abstime=0xffaf48b3a2a8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf30020ed0, timeoutMilliseconds=500) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (Thread 0xffaf48ec61c0 (LWP 33)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48ec50f8, mutex=0xffaf3c00cdf0, cond=0xffaf3c00ce20) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf3c00ce20, mutex=0xffaf3c00cdf0, abstime=0xffaf48ec50f8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf3c00cdf0, timeoutMilliseconds=12000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000ffaf48ec5910 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0xffaf490c71c0 (LWP 32)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf3800d0b0, cond=0xffaf3800d0e0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf3800d0e0, mutex=0xffaf3800d0b0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf3800d0b0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342f150 in ?? ()
#6  0x0000ffaf534302b0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 5 (Thread 0xffaf491de1c0 (LWP 29)):
#0  0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x0000ffaf50fa86ac in SignalHandlerLoop (arg=0xab11907743d0) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3dd9f) at pthread_create.c:463
#3  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 4 (Thread 0xffaf514911c0 (LWP 28)):
#0  0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x0000ffaf543b0b60 in __new_sem_wait_slow (sem=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x0000ffaf53c1c130 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:891
#6  0x0000ffaf53bf6e8c in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1202
#7  0x0000ffaf53bf6d38 in start_wrapper (data=0xab118feeff20) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#8  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e01f) at pthread_create.c:463
#9  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 3 (Thread 0xffaf516921c0 (LWP 27)):
#0  0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000ffaf53e125a0 in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
#2  ds_ipc_poll (poll_handles_data=0xffaf4c0032d0, poll_handles_data_len=1, timeout_ms=4294967295, callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1096
#3  0x0000ffaf53e0fb4c in ds_ipc_stream_factory_get_next_available_stream (callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
#4  0x0000ffaf53e0e3c4 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
#5  0x0000ffaf53e11930 in ep_rt_thread_mono_start_func (data=0xab118fed3d40) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1356
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e16f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 2 (Thread 0xffaf533ff1c0 (LWP 26)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf53e47768 <lock>, cond=0xffaf53e47798 <work_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf53e47798 <work_cond>, mutex=0xffaf53e47768 <lock>) at pthread_cond_wait.c:655
#3  0x0000ffaf53c97b5c in mono_os_cond_wait (cond=0xffaf53e477c0 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e10f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xffaf543edff0 (LWP 25)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xab119041d6c0, cond=0xab119041d6f0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xab119041d6f0, mutex=0xab119041d6c0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xab119041d6c0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342b0e0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Report

Build	Definition	Test	Pull Request
442524	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#93642
442020	dotnet/runtime	ComInterfaceGenerator.Unit.Tests.WorkItemExecution	#93373
2293756	dotnet-runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution	#34502
441666	dotnet/runtime	ComInterfaceGenerator.Unit.Tests.WorkItemExecution	#93373
441292	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#91800
440443	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#93582
440204	dotnet/runtime	System.IO.Pipelines.Tests.WorkItemExecution	#77359
439954	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#92881
439461	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93545
438315	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#92881
436469	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#93408
436470	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#93390
435859	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93280
435814	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#93380
435488	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#93182
435309	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#93182
435247	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93369
434561	dotnet/runtime	JSImportGenerator.Unit.Tests.WorkItemExecution	#93282
433593	dotnet/runtime	System.Numerics.Tensors.Tests.WorkItemExecution	#93296
433363	dotnet/runtime	System.Numerics.Tensors.Tests.WorkItemExecution	#93296
433289	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#93292
433207	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#93289
432794	dotnet/runtime	System.Net.Security.Tests.WorkItemExecution
432457	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93259
432361	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution
432035	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#93182
431388	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#93191
431038	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#93182
431005	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#93180
430413	dotnet/runtime	Microsoft.Extensions.Configuration.Binder.SourceGeneration.Tests.WorkItemExecution
429831	dotnet/runtime	System.ComponentModel.EventBasedAsync.Tests.WorkItemExecution	#93112
428844	dotnet/runtime	System.Text.RegularExpressions.Tests.RegexMatchTests.Match_ExcessPrefix	#93078
428746	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#92980
428612	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution	#93006
428315	dotnet/runtime	JSImportGenerator.Unit.Tests.WorkItemExecution	#93025
428288	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93053
428216	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93049
427970	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92974
427728	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#93027
427388	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#93010
2283304	dotnet-runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#33850
426948	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#79432
426756	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#92980
426746	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution
426427	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#92967
2282611	dotnet-runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#33850
425523	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92864
424117	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92863
423438	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution
423068	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#92829
422545	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92057
422012	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92579
420717	dotnet/runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#92714
420222	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#91633
420178	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92579
420129	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#89587
419941	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92674
419275	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#86787
2276547	dotnet-runtime	Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests.WorkItemExecution	#33850
418883	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#89587
418864	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92578
418525	dotnet/runtime	PayloadGroup0.WorkItemExecution	#92503
418328	dotnet/runtime	IntrinsicsInSystemPrivateCoreLib.Tests.WorkItemExecution	#92503
417623	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#92337
417573	dotnet/runtime	Microsoft.Extensions.Hosting.Tests.ConsoleLifetimeExitTests.EnsureEnvironmentExitCode	#92465
417063	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#92503
416958	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92534
415444	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92410
415389	dotnet/runtime	JSImportGenerator.Unit.Tests.WorkItemExecution
415330	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92457
415290	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#92454
415255	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#92451
414574	dotnet/runtime	JSImportGenerator.Unit.Tests.WorkItemExecution	#90074
414177	dotnet/runtime	PayloadGroup0.WorkItemExecution
413933	dotnet/runtime	LibraryImportGenerator.Unit.Tests.WorkItemExecution	#92287
413702	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92385
413187	dotnet/runtime	Microsoft.Extensions.Http.Tests.WorkItemExecution	#92354
412471	dotnet/runtime	System.Text.RegularExpressions.Tests.WorkItemExecution	#92287
412440	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution	#90436
412090	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	#90436
411570	dotnet/runtime	PayloadGroup0.WorkItemExecution
410140	dotnet/runtime	System.Linq.Queryable.Tests.WorkItemExecution	#90436

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
5	17	82

Known issue validation

Build: 🔎
Result validation: ⚠️ Provided build not found. Provide a valid build in the "Build: 🔎" line.
Validation performed at: 9/1/2023 6:28:52 PM UTC

Tagging subscribers to this area: @dotnet/area-extensions-logging
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Error Blob

Unsure if this is a known issue or not. I didn't find an open with this method failing in the callstack. Feel free to close as dup if you find one.

{
  "ErrorMessage": "in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": true
}

Reproduction Steps

• Affected 7.0 PR: [release/7.0-staging] Reset OOB packages from 7.0.7 #86300
• Affected test: Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests
• Job: Libraries Test Run release mono Linux arm64 Debug
• Job link: https://dev.azure.com/dnceng-public/public/_build/results?buildId=304452&view=logs&j=3f95ca23-483b-5dc9-4b05-6bd24fb925c2&t=0c1dd9d1-4ef7-54d5-2d62-e4ab330605ca&l=56
• Queue: (Ubuntu.1804.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8
• Full callstack:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0xffaf543edff0 (LWP 25) "dotnet" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0xffaf533ff1c0 (LWP 26) "SGen worker" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0xffaf516921c0 (LWP 27) "dotnet" 0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  4    Thread 0xffaf514911c0 (LWP 28) "Finalizer" 0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0xffaf491de1c0 (LWP 29) "dotnet" 0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
  6    Thread 0xffaf490c71c0 (LWP 32) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  7    Thread 0xffaf48ec61c0 (LWP 33) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  8    Thread 0xffaf48b3b1c0 (LWP 34) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
  9    Thread 0xffaf48ada1c0 (LWP 35) ".NET ThreadPool" mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
  10   Thread 0xffaf488861c0 (LWP 36) ".NET Long Runni" 0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  11   Thread 0xffaf486851c0 (LWP 37) ".NET Long Runni" 0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
  12   Thread 0xffaf276891c0 (LWP 39) ".NET ThreadPool" 0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142

Thread 12 (Thread 0xffaf276891c0 (LWP 39)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf276883b8) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf27688340, mutex=0xffaf380c8520, cond=0xffaf27688390) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, abstime=0xffaf27688340) at pthread_cond_wait.c:667
#3  0x0000ffaf53c45e98 in mono_os_cond_timedwait (cond=0xffaf27688390, mutex=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75
#4  0x0000ffaf53c4b494 in mono_coop_cond_timedwait (cond=0xffaf27688390, mutex=<optimized out>, timeout_ms=20000) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:103
#5  mono_lifo_semaphore_timed_wait (semaphore=0xffaf380c8520, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48
#6  0x0000ffaf48aea098 in ?? ()
#7  0x0000ffaf53ba6f28 in ves_icall_System_Threading_Thread_YieldInternal () at /__w/1/s/src/mono/mono/metadata/icall.c:7185
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 11 (Thread 0xffaf486851c0 (LWP 37)):
#0  0x0000ffaf543b2d5c in __waitpid (pid=<optimized out>, stat_loc=0xffaf4867e070, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ffaf53d8706c in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xffaf4867ead0, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ffaf53d455ac in mono_handle_native_crash (signal=0xffaf53b47770 "SIGSEGV", mctx=0xffaf4867ead0, info=0xffaf4867ee30) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ffaf53cafa80 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xffaf4867ee30, context=0xffaf4867eeb0, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3774
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ffaf2789be98 in ?? ()
#8  0x0000ffaf5347b3a8 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 10 (Thread 0xffaf488861c0 (LWP 36)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf2800e9bc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf2800e960, cond=0xffaf2800e990) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf2800e990, mutex=0xffaf2800e960) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf2800e960) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000000000000002 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 9 (Thread 0xffaf48ada1c0 (LWP 35)):
#0  mono_arch_flush_icache (code=0xffaf269b60c8 "\375{\275\251\375\003", size=172) at /__w/1/s/src/mono/mono/mini/mini-arm64.c:2079
#1  0x0000ffaf53ca673c in mono_codegen (cfg=0xffaf357a7990) at /__w/1/s/src/mono/mono/mini/mini.c:2212
#2  0x0000ffaf53ca8da0 in mini_method_compile (method=<optimized out>, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3935
#3  0x0000ffaf53caa0b8 in mono_jit_compile_method_inner (method=0xffaf34f93480, opt=172, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini.c:4129
#4  0x0000ffaf53caeba0 in mono_jit_compile_method_with_opt (method=0xffaf34f93480, opt=374417919, jit_only=0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2708
#5  jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2763
#6  jit_compile_method_with_opt (params=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2779
#7  0x0000ffaf53cae018 in mono_jit_compile_method (method=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2798
#8  0x0000ffaf53d47a18 in common_call_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", m=0xffaf34f93480, vt=0xffaf3540ef10, vtable_slot=<optimized out>, error=0xffaf48ad5930) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618
#9  0x0000ffaf53d48788 in mono_vcall_trampoline (regs=0xffaf48ad59e0, code=0xffaf269b4500 "", slot=1, tramp=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:840
#10 0x0000ffaf53920504 in ?? ()
#11 0x0000ffaf5368b8d0 in ?? ()
#12 0x0000ffaf5368cec8 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind further

Thread 8 (Thread 0xffaf48b3b1c0 (LWP 34)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x0, expected=0, futex_word=0xffaf30020f28) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48b3a2a8, mutex=0xffaf30020ed0, cond=0xffaf30020f00) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf30020f00, mutex=0xffaf30020ed0, abstime=0xffaf48b3a2a8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf30020ed0, timeoutMilliseconds=500) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000000000000001 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Thread 7 (Thread 0xffaf48ec61c0 (LWP 33)):
#0  0x0000ffaf543ae5bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0xffaf00000000, expected=0, futex_word=0xffaf3c00ce48) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0xffaf48ec50f8, mutex=0xffaf3c00cdf0, cond=0xffaf3c00ce20) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0xffaf3c00ce20, mutex=0xffaf3c00cdf0, abstime=0xffaf48ec50f8) at pthread_cond_wait.c:667
#3  0x0000ffaf50fa945c in SystemNative_LowLevelMonitor_TimedWait (monitor=0xffaf3c00cdf0, timeoutMilliseconds=12000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000ffaf48ae0894 in ?? ()
#5  0x0000ffaf490d71d0 in ?? ()
#6  0x0000ffaf48ec5910 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 6 (Thread 0xffaf490c71c0 (LWP 32)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf3800d10c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf3800d0b0, cond=0xffaf3800d0e0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf3800d0e0, mutex=0xffaf3800d0b0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xffaf3800d0b0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342f150 in ?? ()
#6  0x0000ffaf534302b0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 5 (Thread 0xffaf491de1c0 (LWP 29)):
#0  0x0000ffaf543b1ac0 in __libc_read (fd=<optimized out>, buf=0xffaf491dd9a7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1  0x0000ffaf50fa86ac in SignalHandlerLoop (arg=0xab11907743d0) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3dd9f) at pthread_create.c:463
#3  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 4 (Thread 0xffaf514911c0 (LWP 28)):
#0  0x0000ffaf543b0a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0xffaf53e387d8 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  do_futex_wait (sem=sem@entry=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
#2  0x0000ffaf543b0b60 in __new_sem_wait_slow (sem=0xffaf53e387d8 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
#3  0x0000ffaf53c1c130 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
#4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
#5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:891
#6  0x0000ffaf53bf6e8c in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1202
#7  0x0000ffaf53bf6d38 in start_wrapper (data=0xab118feeff20) at /__w/1/s/src/mono/mono/metadata/threads.c:1264
#8  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e01f) at pthread_create.c:463
#9  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 3 (Thread 0xffaf516921c0 (LWP 27)):
#0  0x0000ffaf54088ef8 in __GI___poll (fds=0xffaf4c003ae0, nfds=281128488714676, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000ffaf53e125a0 in ipc_poll_fds (fds=<optimized out>, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470
#2  ds_ipc_poll (poll_handles_data=0xffaf4c0032d0, poll_handles_data_len=1, timeout_ms=4294967295, callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1096
#3  0x0000ffaf53e0fb4c in ds_ipc_stream_factory_get_next_available_stream (callback=0xffaf53e11950 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:395
#4  0x0000ffaf53e0e3c4 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:127
#5  0x0000ffaf53e11930 in ep_rt_thread_mono_start_func (data=0xab118fed3d40) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1356
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e16f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 2 (Thread 0xffaf533ff1c0 (LWP 26)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xffaf53e477c0 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xffaf53e47768 <lock>, cond=0xffaf53e47798 <work_cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xffaf53e47798 <work_cond>, mutex=0xffaf53e47768 <lock>) at pthread_cond_wait.c:655
#3  0x0000ffaf53c97b5c in mono_os_cond_wait (cond=0xffaf53e477c0 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
#4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
#5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
#6  0x0000ffaf543a8088 in start_thread (arg=0xfffff0f3e10f) at pthread_create.c:463
#7  0x0000ffaf540920cc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xffaf543edff0 (LWP 25)):
#0  0x0000ffaf543ae2a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xab119041d71c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0xab119041d6c0, cond=0xab119041d6f0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0xab119041d6f0, mutex=0xab119041d6c0) at pthread_cond_wait.c:655
#3  0x0000ffaf50fa92f8 in SystemNative_LowLevelMonitor_Wait (monitor=0xab119041d6c0) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4  0x0000ffaf490d79a4 in ?? ()
#5  0x0000ffaf5342b0e0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Author:	carlossanlop
Assignees:	-
Labels:	arch-arm64, blocking-clean-ci, area-VM-meta-mono, runtime-mono, area-Extensions-Logging, Known Build Error
Milestone:	-

[carlossanlop]

There are variants that KnownBuild is not collecting and tagging, so I think this issue is much more impactful that we are seeing. Here is an example:

cc @lewing @steveisok

• PR: [release/8.0] Fix diagnostic creation to ensure including the URI #91145
• Affected test: System.Linq
• Queue: Libraries Test Run release mono_interpreter linux x64 Debug
• Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=386567&view=logs&j=9310185d-988f-5930-4a09-6918c1bb36fb&t=70410075-93d6-55d0-0186-00493ab8b56c&l=55
• Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-91145-merge-0704fab9cc0c4c5580/System.Linq.Queryable.Tests/1/console.e9ef3c29.log?helixlogtype=result

Output (expand)

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Linq.Queryable.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Linq.Queryable.Tests (found 785 test cases)
  Starting:    System.Linq.Queryable.Tests (parallel test collections = on, max threads = 2)

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x7f17cef12771 - Unknown
	0x7f17ceeb9abe - Unknown
	0x7f17cee21b31 - Unknown
	0x7f17cf74d140 - Unknown
	0x7f17cef15cc9 - Unknown
	0x7f17cef14296 - Unknown
	0x7f17cf041b97 - Unknown
	0x7f17cf04a74b - Unknown
	0x7f17cf001d53 - Unknown
	0x7f17cf00c36b - Unknown
	0x7f17cef2695f - Unknown
	0x7f17cef25115 - Unknown
	0x7f17cef1699a - Unknown
	0x7f17cef14296 - Unknown
	0x7f17cf041b97 - Unknown
	0x7f17cf059db4 - Unknown
	0x7f17cf741ea7 - Unknown
	0x7f17cf330a2f - Unknown

=================================================================
	External Debugger Dump:
=================================================================
[New LWP 23]
[New LWP 24]
[New LWP 25]
[New LWP 26]
[New LWP 29]
[New LWP 30]
[New LWP 31]
[New LWP 32]
[New LWP 33]
[New LWP 34]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so.dbg]
DW_FORM_rnglistx index pointing outside of .debug_rnglists offset array [in module /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libSystem.Native.so.dbg]
futex_wait_cancelable (private=0, expected=0, futex_word=0x558565f264d4) at ../sysdeps/nptl/futex-internal.h:186
186	../sysdeps/nptl/futex-internal.h: No such file or directory.
  Id   Target Id                                        Frame 
* 1    Thread 0x7f17cf230740 (LWP 22) "dotnet"          futex_wait_cancelable (private=0, expected=0, futex_word=0x558565f264d4) at ../sysdeps/nptl/futex-internal.h:186
  2    Thread 0x7f17ce3ff700 (LWP 23) "SGen worker"     futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17cf113d68 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
  3    Thread 0x7f17cc601700 (LWP 24) ".NET EventPipe"  0x00007f17cf32496f in __GI___poll (fds=0x7f17c8002890, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
  4    Thread 0x7f17cc400700 (LWP 25) "Finalizer"       futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf104f50 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
  5    Thread 0x7f17c558d700 (LWP 26) ".NET SigHandler" __libc_read (nbytes=1, buf=0x7f17c558cee7, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
  6    Thread 0x7f17c54f0700 (LWP 29) ".NET Long Runni" futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17b4248df0) at ../sysdeps/nptl/futex-internal.h:186
  7    Thread 0x7f17c51ee700 (LWP 30) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c51ed668, clockid=-987834928, expected=0, futex_word=0x7f17b80a7250) at ../sysdeps/nptl/futex-internal.h:323
  8    Thread 0x7f17c5268700 (LWP 31) ".NET TP Gate"    futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17ac000bc0) at ../sysdeps/nptl/futex-internal.h:323
  9    Thread 0x7f17c490b700 (LWP 32) ".NET TP Worker"  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c490a6e8, clockid=-997153184, expected=0, futex_word=0x7f17c490a768) at ../sysdeps/nptl/futex-internal.h:323
  10   Thread 0x7f17c4609700 (LWP 33) ".NET Long Runni" 0x00007f17cf2fc747 in __GI___wait4 (pid=35, stat_loc=0x7f17c4605080, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
  11   Thread 0x7f17c4307700 (LWP 34) ".NET Long Runni" futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf0fef10 <suspend_semaphore>) at ../sysdeps/nptl/futex-internal.h:323

Thread 11 (Thread 0x7f17c4307700 (LWP 34) ".NET Long Runni"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf0fef10 <suspend_semaphore>) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7f17cf0fef10 <suspend_semaphore>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007f17cf74b278 in __new_sem_wait_slow (sem=0x7f17cf0fef10 <suspend_semaphore>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007f17cefcacc8 in mono_os_sem_timedwait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefca965 in mono_threads_wait_pending_operations () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cf08d546 in unified_suspend_stop_world () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cf08d24d in sgen_client_stop_world () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cf0ae2cb in sgen_stop_world () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf0ab08e in sgen_perform_collection () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf0aaff3 in sgen_ensure_free_space () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf0a647d in sgen_alloc_obj_nolock () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cf0a6999 in sgen_alloc_obj () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007f17cf08f0d6 in mono_gc_alloc_obj () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007f17cef1c01a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#14 0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#15 0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#16 0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#17 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f17c4609700 (LWP 33) ".NET Long Runni"):
#0  0x00007f17cf2fc747 in __GI___wait4 (pid=35, stat_loc=0x7f17c4605080, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1  0x00007f17cef128a2 in mono_dump_native_crash_info () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007f17ceeb9abe in mono_handle_native_crash () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007f17cee21b31 in mono_sigsegv_signal_handler_debug () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  <signal handler called>
#5  0x00007f17cef15cc9 in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf04a74b in mono_runtime_try_invoke_byrefs () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf001d53 in ves_icall_InternalInvoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf00c36b in ves_icall_InternalInvoke_raw () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cef2695f in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007f17cef25115 in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#14 0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#15 0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#16 0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#17 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f17c490b700 (LWP 32) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c490a6e8, clockid=-997153184, expected=0, futex_word=0x7f17c490a768) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7f17c490a6e8, clockid=-997153184, mutex=0x7f17b40970b0, cond=0x7f17c490a740) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7f17c490a740, mutex=0x7f17b40970b0, abstime=0x7f17c490a6e8) at pthread_cond_wait.c:656
#3  0x00007f17cefc39f3 in mono_os_cond_timedwait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefc8899 in mono_lifo_semaphore_timed_wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef268d7 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef25115 in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#12 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f17c5268700 (LWP 31) ".NET TP Gate"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17ac000bc0) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7f17ac000bc0, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007f17cf74b278 in __new_sem_wait_slow (sem=0x7f17ac000bc0, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007f17cefca7c8 in mono_thread_info_wait_for_resume () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefd02c1 in mono_threads_exit_gc_safe_region_unbalanced_internal () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250de in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f17c51ee700 (LWP 30) ".NET TP Worker"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f17c51ed668, clockid=-987834928, expected=0, futex_word=0x7f17b80a7250) at ../sysdeps/nptl/futex-internal.h:323
#1  __pthread_cond_wait_common (abstime=0x7f17c51ed668, clockid=-987834928, mutex=0x7f17b80a7200, cond=0x7f17b80a7228) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x7f17b80a7228, mutex=0x7f17b80a7200, abstime=0x7f17c51ed668) at pthread_cond_wait.c:656
#3  0x00007f17ce82ca97 in SystemNative_LowLevelMonitor_TimedWait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007f17cef268d7 in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250cf in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f17c54f0700 (LWP 29) ".NET Long Runni"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17b4248df0) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f17b4248da0, cond=0x7f17b4248dc8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7f17b4248dc8, mutex=0x7f17b4248da0) at pthread_cond_wait.c:638
#3  0x00007f17ce82c937 in SystemNative_LowLevelMonitor_Wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007f17cef2688e in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250cf in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf059db4 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f17c558d700 (LWP 26) ".NET SigHandler"):
#0  __libc_read (nbytes=1, buf=0x7f17c558cee7, fd=7) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __libc_read (fd=7, buf=0x7f17c558cee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007f17ce82c086 in SignalHandlerLoop () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#3  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#4  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f17cc400700 (LWP 25) "Finalizer"):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x7f17cf104f50 <finalizer_sem>) at ../sysdeps/nptl/futex-internal.h:323
#1  do_futex_wait (sem=sem@entry=0x7f17cf104f50 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:112
#2  0x00007f17cf74b278 in __new_sem_wait_slow (sem=0x7f17cf104f50 <finalizer_sem>, abstime=0x0, clockid=0) at sem_waitcommon.c:184
#3  0x00007f17cf081008 in finalizer_thread () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cf059d09 in start_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f17cc601700 (LWP 24) ".NET EventPipe"):
#0  0x00007f17cf32496f in __GI___poll (fds=0x7f17c8002890, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f17cefa455e in ds_ipc_poll () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#2  0x00007f17cefa1288 in ds_ipc_stream_factory_get_next_available_stream () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#3  0x00007f17cef9f7af in server_thread () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cefa39c1 in ep_rt_thread_mono_start_func () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#6  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f17ce3ff700 (LWP 23) "SGen worker"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f17cf113d68 <work_cond+40>) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f17cf113d18 <lock>, cond=0x7f17cf113d40 <work_cond>) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x7f17cf113d40 <work_cond>, mutex=0x7f17cf113d18 <lock>) at pthread_cond_wait.c:638
#3  0x00007f17cf0e8ac3 in thread_func () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#4  0x00007f17cf741ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#5  0x00007f17cf330a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f17cf230740 (LWP 22) "dotnet"):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x558565f264d4) at ../sysdeps/nptl/futex-internal.h:186
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x558565f26480, cond=0x558565f264a8) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0x558565f264a8, mutex=0x558565f26480) at pthread_cond_wait.c:638
#3  0x00007f17ce82c937 in SystemNative_LowLevelMonitor_Wait () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0//libSystem.Native.so
#4  0x00007f17cef2688e in do_icall () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#5  0x00007f17cef250cf in do_icall_wrapper () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#6  0x00007f17cef1699a in mono_interp_exec_method () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#7  0x00007f17cef14296 in interp_runtime_invoke () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#8  0x00007f17cf041b97 in mono_runtime_invoke_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#9  0x00007f17cf048808 in mono_runtime_exec_main_checked () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#10 0x00007f17cee7a51f in mono_jit_exec () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#11 0x00007f17cee7cd8b in mono_main () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#12 0x00007f17ceef0ad7 in monovm_execute_assembly () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libcoreclr.so
#13 0x00007f17cf14c778 in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#14 0x00007f17cf17adea in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#15 0x00007f17cf17b0df in ?? () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#16 0x00007f17cf17c082 in corehost_main () from /root/helix/work/correlation/shared/Microsoft.NETCore.App/8.0.0/libhostpolicy.so
#17 0x00007f17cf1f6afc in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#18 0x00007f17cf1f1c73 in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#19 0x00007f17cf1ee0ec in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#20 0x00007f17cf1ed1cb in ?? () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#21 0x00007f17cf1e6115 in hostfxr_main_startupinfo () from /root/helix/work/correlation/host/fxr/8.0.0/libhostfxr.so
#22 0x0000558562e2e312 in ?? ()
#23 0x0000558562e2e835 in ?? ()
#24 0x00007f17cf258d0a in __libc_start_main (main=0x558562e2e750, argc=18, argv=0x7ffcc15b9eb8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffcc15b9ea8) at ../csu/libc-start.c:308
#25 0x0000558562e21979 in ?? ()
[Inferior 1 (process 22) detached]

=================================================================
	Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7f17cef15cc9):0x7f17cef15cb9  1d 00 49 89 db ff 24 c1 49 8b 44 24 08 48 8b 00  ..I...$.I.D$.H..
0x7f17cef15cc9  48 8b 70 18 48 8d 3d de 48 e6 ff 31 c0 e8 05 9f  H.p.H.=.H..1....
0x7f17cef15cd9  09 00 48 8d 3d 48 90 e4 ff be 17 0f 00 00 e8 c4  ..H.=H..........
0x7f17cef15ce9  a3 09 00 0f 1f 40 00 4c 89 e6 4c 89 db 48 83 c3  .....@.L..L..H..

=================================================================
	Managed Stacktrace:
=================================================================
	  at <unknown> <0xffffffff>
	  at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x00096>
	  at System.Reflection.MethodBaseInvoker:InterpretedInvoke_Method <0x0003c>
	  at System.Reflection.MethodBaseInvoker:InvokeWithNoArgs <0x0006e>
	  at System.Reflection.RuntimeMethodInfo:Invoke <0x000fe>
	  at System.Reflection.MethodBase:Invoke <0x0002e>
	  at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x00030>
	  at <<InvokeTestMethodAsync>b__1>d:MoveNext <0x00242>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__1 <0x00060>
	  at <AggregateAsync>d__4:MoveNext <0x0007a>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x00074>
	  at <>c__DisplayClass48_0:<InvokeTestMethodAsync>b__0 <0x00082>
	  at <RunAsync>d__9:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at <InvokeTestMethodAsync>d__48:MoveNext <0x00122>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x00074>
	  at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x000b8>
	  at <<RunAsync>b__47_0>d:MoveNext <0x003ba>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestInvoker`1:<RunAsync>b__47_0 <0x00060>
	  at <RunAsync>d__10`1:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at Xunit.Sdk.TestInvoker`1:RunAsync <0x0003a>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x000a8>
	  at <InvokeTestAsync>d__4:MoveNext <0x00178>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x00074>
	  at <>c__DisplayClass43_0:<RunAsync>b__0 <0x0002c>
	  at <RunAsync>d__10`1:MoveNext <0x0005e>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00074>
	  at <RunAsync>d__43:MoveNext <0x002aa>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCaseRunner:RunTestAsync <0x00104>
	  at <RunAsync>d__19:MoveNext <0x001ee>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCase:RunAsync <0x0006a>
	  at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x00076>
	  at <RunTestCasesAsync>d__32:MoveNext <0x000fe>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x00060>
	  at <RunAsync>d__31:MoveNext <0x00114>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x000a2>
	  at <RunTestMethodsAsync>d__38:MoveNext <0x003a4>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x00060>
	  at <RunAsync>d__37:MoveNext <0x0020c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestClassRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x000aa>
	  at <RunTestClassesAsync>d__28:MoveNext <0x001a4>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x00060>
	  at <RunAsync>d__27:MoveNext <0x0020c>
	  at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0007a>
	  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0000e>
	  at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x00060>
	  at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x00076>
	  at <>c__DisplayClass14_2:<RunTestCollectionsAsync>b__2 <0x00092>
	  at System.Threading.Tasks.Task`1:InnerInvoke <0x00052>
	  at <>c:<.cctor>b__281_0 <0x0001c>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x001f0>
	  at System.Threading.Tasks.Task:ExecuteEntry <0x000b6>
	  at <>c:<.cctor>b__8_0 <0x0001c>
	  at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x00042>
	  at <>c__DisplayClass11_0:<WorkerThreadProc>b__0 <0x00056>
	  at System.Action`1:invoke_void_T <0x00080>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Object:lambda_method2 <0x0004a>
	  at Xunit.Sdk.ExecutionContextHelper:Run <0x00034>
	  at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x0016c>
	  at <>c:<QueueUserWorkItem>b__5_0 <0x00038>
	  at System.Threading.Tasks.Task:InnerInvoke <0x00092>
	  at <>c:<.cctor>b__281_0 <0x0001c>
	  at System.Threading.ExecutionContext:RunInternal <0x000f0>
	  at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x001f0>
	  at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x00086>
	  at <>c:<.cctor>b__10_0 <0x00020>
	  at System.Object:runtime_invoke_direct_void__this__ <0x0022c>
	  at <unknown> <0x00000>
=================================================================
./RunTests.sh: line 168:    22 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Linq.Queryable.Tests.runtimeconfig.json --depsfile System.Linq.Queryable.Tests.deps.json xunit.console.dll System.Linq.Queryable.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Fri 25 Aug 2023 11:45:42 PM UTC ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.