Open
Description
openedon May 25, 2022
Failures 7/5-8/5 (incl. PRs) based on Runfo in last 30 days:
Day | Run | Failure |
---|---|---|
8/4 | PR #73174 - 1923945 | Crash dump |
8/4 | Rolling run 1923908 | Crash dump |
fail: Application has finished with exit code 1 but 0 was expected - likely infra hiccup | ||
8/3 | PR #73265 - 1921135 | Crash dump |
8/2 | PR #72443 - 1918306 | Crash dump |
8/2 | PR #72851 - 1917737 | Crash dump |
8/2 | PR #72143 - 1917321 | Crash dump |
No clear error - likely infra hiccup | ||
2x Segmentation fault - most likely caused by the PR | ||
7/29 | PR #73009 - 1912477 | Crash dump |
2x Segmentation fault - most likely caused by the PR | ||
2x Segmentation fault - most likely caused by the PR | ||
7/28 | PR #67272 - 1909374 | Unhandled Exception |
7/28 | PR #72762 - 1908782 | Unhandled Exception |
7/28 | PR #69542 - 1908613 | Crash dump |
7/26 | PR #71661 - 1905330 | Unhandled Exception |
7/26 | Rolling run 1904930 | Crash dump |
7/24 | PR #72448 - 1900699 | Crash dump |
7/24 | PR #72732 - 1900156 | Crash dump |
7/22 | PR #72672 - 1897479 | Unhandled Exception |
7/20 | PR #72482 - 1893721 | Crash dump |
7/20 | PR #72529 - 1892945 | Crash dump |
7/19 | PR #72394 - 1891289 | Crash dump |
7/19 | PR #72452 - 1891146 | Unhandled Exception |
7/16 | PR #71827 - 1885040 | Crash dump |
7/16 | PR #72275 - 1885005 | Crash dump |
7/15 | PR #72021 - 1882959 | Crash dump |
4/5-7/14 | Logs not available |
Unhandled Exception
Example: Console log
16:13:46.844 Running test: JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd
Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation_REF op)
at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op)
at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process()
at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events)
at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation_REF op)
at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op)
at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
at System.Net.Sockets.SocketAsyncContext.AsyncOperation.Process()
at System.Net.Sockets.SocketAsyncContext.HandleEvents(SocketEvents events)
at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
at System.Threading.Thread.StartCallback()
Crash dump
Observed in https://dev.azure.com/dnceng/public/_build/results?buildId=1789473
Relevant log snippet:
22:49:29.737 Running test: JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd
=================================================================
Native Crash Reporting
=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
=================================================================
Native stacktrace:
=================================================================
0x7fa4c2a98722 - Unknown
0x7fa4c2a39a2e - Unknown
0x7fa4c2996d71 - Unknown
0x7fa4c4e52980 - Unknown
0x7fa4c29e5494 - Unknown
0x7fa4bed63df3 - Unknown
=================================================================
External Debugger Dump:
=================================================================
[New LWP 26773]
[New LWP 26774]
[New LWP 26775]
[New LWP 26776]
[New LWP 26781]
[New LWP 26785]
[New LWP 26786]
[New LWP 26787]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
warning: the debug information found in "/datadisks/disk1/work/B4770A02/p/libcoreclr.so.dbg" does not match "/datadisks/disk1/work/B4770A02/p/libcoreclr.so" (CRC mismatch).
0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88 ../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
Id Target Id Frame
* 1 Thread 0x7fa4c5479740 (LWP 26772) "corerun" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
2 Thread 0x7fa4c13ff700 (LWP 26773) "SGen worker" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa4c40d95c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
3 Thread 0x7fa4be6cc700 (LWP 26774) "corerun" 0x00007fa4c4224bb9 in __GI___poll (fds=0x7fa4b8002d50, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
4 Thread 0x7fa4be4cb700 (LWP 26775) "Finalizer" 0x00007fa4c4e507c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa4c40ca530) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
5 Thread 0x7fa4b63fe700 (LWP 26776) "corerun" 0x00007fa4c4e51474 in __libc_read (fd=7, buf=0x7fa4b63fdee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
6 Thread 0x7fa4a8194700 (LWP 26781) ".NET Sockets" 0x00007fa4c4231947 in epoll_wait (epfd=13, events=0x5627021f5630, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
7 Thread 0x7fa4a7d8d700 (LWP 26785) ".NET ThreadPool" 0x00007fa4c4e4dfb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa4a7d8c6f0, expected=0, futex_word=0x7fa4a7d8c828) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
8 Thread 0x7fa4a7b8c700 (LWP 26786) ".NET ThreadPool" 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa49c002c90) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
9 Thread 0x7fa4a7b4b700 (LWP 26787) ".NET ThreadPool" 0x00007fa4c4e5232a in __waitpid (pid=26788, stat_loc=0x7fa4a7b491c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
Thread 9 (Thread 0x7fa4a7b4b700 (LWP 26787)):
#0 0x00007fa4c4e5232a in __waitpid (pid=26788, stat_loc=0x7fa4a7b491c0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1 0x00007fa4c2a98867 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#2 0x00007fa4c2a39a2e in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#3 0x00007fa4c2996d71 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4 <signal handler called>
#5 0x00007fa4c29e5494 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6 0x00007fa4bed63df3 in System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_TResult_INT_SetExistingTaskResult_System_Threading_Tasks_Task_1_TResult_INT_TResult_INT () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7 0x00005627026707f8 in ?? ()
#8 0x00007fa4c143f198 in ?? ()
#9 0x00007fa4c143f620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 8 (Thread 0x7fa4a7b8c700 (LWP 26786)):
#0 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa49c002c90) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa49c002c40, cond=0x7fa49c002c68) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x7fa49c002c68, mutex=0x7fa49c002c40) at pthread_cond_wait.c:655
#3 0x00007fa4bcf3d1fc in SystemNative_LowLevelMonitor_Wait (monitor=0x7fa49c002c40) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155
#4 0x00007fa4bf0ee1e6 in wrapper_managed_to_native_Interop_Sys_LowLevelMonitor_Wait_intptr () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#5 0x00007fa4bea252df in System_Threading_WaitSubsystem_ThreadWaitInfo_Wait_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#6 0x00007fa4bea2688e in System_Threading_WaitSubsystem_WaitableObject_Wait_Locked_System_Threading_WaitSubsystem_ThreadWaitInfo_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7 0x00007fa4bea265fb in System_Threading_WaitSubsystem_WaitableObject_Wait_System_Threading_WaitSubsystem_ThreadWaitInfo_int_bool_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#8 0x00007fa4a7b8bbb0 in ?? ()
#9 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7fa4a7d8d700 (LWP 26785)):
#0 0x00007fa4c4e4dfb9 in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7fa4a7d8c6f0, expected=0, futex_word=0x7fa4a7d8c828) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1 __pthread_cond_wait_common (abstime=0x7fa4a7d8c7a8, mutex=0x7fa4a000b800, cond=0x7fa4a7d8c800) at pthread_cond_wait.c:533
#2 __pthread_cond_timedwait (cond=0x7fa4a7d8c800, mutex=0x7fa4a000b800, abstime=0x7fa4a7d8c7a8) at pthread_cond_wait.c:667
#3 0x00007fa4c292a474 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4 0x00007fa4c292fb74 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5 0x00007fa4bef1e863 in wrapper_managed_to_native_System_Threading_LowLevelLifoSemaphore_TimedWaitInternal_intptr_int () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#6 0x00007fa4be9fb9cd in System_Threading_LowLevelLifoSemaphore_WaitForSignal_int () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#7 0x00007fa4c1441d08 in ?? ()
#8 0x0000000000000002 in ?? ()
#9 0xfffeffffffffffff in ?? ()
#10 0x0000000000004e20 in ?? ()
#11 0x0000000000000046 in ?? ()
#12 0x00007fa4c410ee30 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#13 0x00007fa4be9fb6db in System_Threading_LowLevelLifoSemaphore_Wait_int_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#14 0x00007fa4c1441998 in ?? ()
#15 0x00007fa4a7d8c9d0 in ?? ()
#16 0x00007fa4a7d8ca08 in ?? ()
#17 0x00007fa4a7d8cdc8 in ?? ()
#18 0x00007fa4c410ee30 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#19 0x00007fa4a7d8ca80 in ?? ()
#20 0x00007fa4bea20b1d in System_Threading_PortableThreadPool_WorkerThread_WorkerThreadStart () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
Thread 6 (Thread 0x7fa4a8194700 (LWP 26781)):
#0 0x00007fa4c4231947 in epoll_wait (epfd=13, events=0x5627021f5630, maxevents=32, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007fa4bcf37f9e in WaitForSocketEventsInner (port=13, buffer=0x5627021f5630, count=0x7fa4a81939d4) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:2717
#2 0x00007fa4bcf37ebf in SystemNative_WaitForSocketEvents (port=13, buffer=0x5627021f5630, count=0x7fa4a81939d4) at /__w/1/s/src/native/libs/System.Native/pal_networking.c:3017
#3 0x00007fa4a8eef485 in wrapper_managed_to_native_Interop_Sys_WaitForSocketEvents_intptr_Interop_Sys_SocketEvent__int_ (param0=13, param1=94725539321392, param2=140345171589588) at /_/src/libraries/System.Collections.Concurrent/src/System/Collections/Concurrent/ConcurrentDictionary.cs:2264
#4 0x00007fa4a8e30f1c in System_Net_Sockets_SocketAsyncEngine_EventLoop () at /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEngine.Unix.cs:184
#5 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fa4b63fe700 (LWP 26776)):
#0 0x00007fa4c4e51474 in __libc_read (fd=7, buf=0x7fa4b63fdee7, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:27
#1 0x00007fa4bcf3c60e in SignalHandlerLoop (arg=0x562702331280) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:323
#2 0x00007fa4c4e476db in start_thread (arg=0x7fa4b63fe700) at pthread_create.c:463
#3 0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 4 (Thread 0x7fa4be4cb700 (LWP 26775)):
#0 0x00007fa4c4e507c6 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa4c40ca530) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1 do_futex_wait (sem=sem@entry=0x7fa4c40ca530, abstime=0x0) at sem_waitcommon.c:111
#2 0x00007fa4c4e508b8 in __new_sem_wait_slow (sem=0x7fa4c40ca530, abstime=0x0) at sem_waitcommon.c:181
#3 0x00007fa4c29011c6 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4 0x00007fa4c28da2ba in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5 0x00007fa4c28da149 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6 0x00007fa4c4e476db in start_thread (arg=0x7fa4be4cb700) at pthread_create.c:463
#7 0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 3 (Thread 0x7fa4be6cc700 (LWP 26774)):
#0 0x00007fa4c4224bb9 in __GI___poll (fds=0x7fa4b8002d50, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007fa4c2b5f84a in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#2 0x00007fa4c2b5cd45 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#3 0x00007fa4c2b5b5a9 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4 0x00007fa4c2b5ec61 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5 0x00007fa4c4e476db in start_thread (arg=0x7fa4be6cc700) at pthread_create.c:463
#6 0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 2 (Thread 0x7fa4c13ff700 (LWP 26773)):
#0 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa4c40d95c8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa4c40d9578, cond=0x7fa4c40d95a0) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x7fa4c40d95a0, mutex=0x7fa4c40d9578) at pthread_cond_wait.c:655
#3 0x00007fa4c297ee73 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4 0x00007fa4c4e476db in start_thread (arg=0x7fa4c13ff700) at pthread_create.c:463
#5 0x00007fa4c423161f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 1 (Thread 0x7fa4c5479740 (LWP 26772)):
#0 0x00007fa4c4e4dad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x562701fd5188) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x562701fd5138, cond=0x562701fd5160) at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x562701fd5160, mutex=0x562701fd5138) at pthread_cond_wait.c:655
#3 0x00007fa4c292a490 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#4 0x00007fa4c28df52a in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#5 0x00007fa4c28df3a8 in mono_w32handle_wait_one () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#6 0x00007fa4c2902bb4 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#7 0x00007fa4c2892b93 in ?? () from /datadisks/disk1/work/B4770A02/p/libcoreclr.so
#8 0x00007fa4bef1d79d in wrapper_managed_to_native_System_Threading_Monitor_Monitor_wait_object_int_bool () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#9 0x00007fa4be9f5986 in System_Threading_Monitor_ObjWait_int_object () from /datadisks/disk1/work/B4770A02/p/System.Private.CoreLib.dll.so
#10 0x0000000000000001 in ?? ()
#11 0x00007fa4c1440a30 in ?? ()
#12 0x0000000000000000 in ?? ()
=================================================================
Basic Fault Address Reporting
=================================================================
Memory around native instruction pointer (0x7fa4c29e5494):0x7fa4c29e5484 57 01 48 8b 40 38 48 8b 80 98 00 00 00 48 8b 38 W.H.@8H......H.8
0x7fa4c29e5494 41 8b 46 10 8d 34 c5 08 00 00 00 e8 6c 91 f1 ff A.F..4......l...
0x7fa4c29e54a4 48 89 04 24 41 83 7e 10 00 0f 8e 80 00 00 00 48 H..$A.~........H
0x7fa4c29e54b4 63 c5 48 89 44 24 10 f7 d8 48 89 44 24 08 48 8d c.H.D$...H.D$.H.
=================================================================
Managed Stacktrace:
=================================================================
=================================================================
JIT/Methodical/Methodical_others/Methodical_others.sh: line 382: 26772 Aborted (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"
Runfo Tracking Issue: methodical_others work item
Build | Definition | Kind | Run Name |
---|
Build Result Summary
Day Hit Count | Week Hit Count | Month Hit Count |
---|---|---|
0 | 0 | 0 |
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment