ATS 8.0.0 coredump in EventIO

[bryancall]

(gdb) bt
#0 0x00002ad3e68e71d7 in raise () from /lib64/libc.so.6
#1 0x00002ad3e68e88c8 in abort () from /lib64/libc.so.6
#2 0x00002ad3e42ebba9 in ink_abort (message_format=0x2ad3e431738c "%s:%d: failed assertion %s") at ../../../../trafficserver/lib/ts/ink_error.cc:99
#3 0x00002ad3e42e8fac in _ink_assert (expression=0x893671 "event_loop", file=0x893640 "../../../../trafficserver/iocore/net/P_UnixNet.h", line=694) at ../../../../trafficserver/lib/ts/ink_assert.cc:37
#4 0x00000000007a7bdb in EventIO::refresh (this=0x2ad42c808200, e=-2147483647) at ../../../../trafficserver/iocore/net/P_UnixNet.h:694
#5 0x00000000007af8de in read_reschedule (nh=0x2ad3ee395cd0, vc=0x2ad42c807f80) at ../../../../trafficserver/iocore/net/UnixNetVConnection.cc:44
#6 0x00000000007b3027 in UnixNetVConnection::readReschedule (this=0x2ad42c807f80, nh=0x2ad3ee395cd0) at ../../../../trafficserver/iocore/net/UnixNetVConnection.cc:1056
#7 0x0000000000789084 in SSLNetVConnection::net_read_io (this=0x2ad42c807f80, nh=0x2ad3ee395cd0, lthread=0x2ad3ee392010) at ../../../../trafficserver/iocore/net/SSLNetVConnection.cc:465
#8 0x00000000007a619a in NetHandler::process_ready_list (this=0x2ad3ee395cd0) at ../../../../trafficserver/iocore/net/UnixNet.cc:395
#9 0x00000000007a6966 in NetHandler::waitForActivity (this=0x2ad3ee395cd0, timeout=60000000) at ../../../../trafficserver/iocore/net/UnixNet.cc:528
#10 0x00000000007db5be in EThread::execute_regular (this=0x2ad3ee392010) at ../../../../trafficserver/iocore/eventsystem/UnixEThread.cc:272
#11 0x00000000007db7ad in EThread::execute (this=0x2ad3ee392010) at ../../../../trafficserver/iocore/eventsystem/UnixEThread.cc:325
#12 0x00000000007da1bf in spawn_thread_internal (a=0x1621f50) at ../../../../trafficserver/iocore/eventsystem/Thread.cc:85
#13 0x00002ad3e5c7adc5 in start_thread () from /lib64/libpthread.so.0
#14 0x00002ad3e69a976d in clone () from /lib64/libc.so.6

[bryancall]

I am not seeing this anymore might be related to #3962

[pbchou]

We are seeing a very similar core dump on assertion failure with 7.1.4. We understand that 7.1.x is no longer supported. So we are not looking for a patch. Would appreciate an assessment on whether our core dump would likely be same the issue as this and thus also corrected in 8.x forwards. We are looking to deploy 9.0.x soon, and my users are looking for this issue to fixed by the upgrade.

Our core dump is reproducible in production by issuing

 traffic_ctl config reload 

on heavily loaded cache. I deployed a debug build (which is not stable for more that some minutes) with instrumentation added to the EventIO class. I can tell that the object instance with event_loop == nullptr has called the stop() before the assertion failure on the call to the refresh() method. So it is still active even after being stopped.

(gdb) bt
#0  0x00002b4f69969377 in raise () from /lib64/libc.so.6
#1  0x00002b4f6996aba8 in abort () from /lib64/libc.so.6
#2  0x00002b4f66fcb4a3 in ink_abort (message_format=0x2b4f66fdd1a4 "%s:%d: failed assertion `%s`") at ink_error.cc:99
#3  0x00002b4f66fc8b6d in _ink_assert (expression=0x867f2f "event_loop", file=0x867f23 "P_UnixNet.h", line=586) at ink_assert.cc:37
#4  0x00000000007aa60d in EventIO::refresh (this=0x2b4f7456b640, e=-2147483647) at P_UnixNet.h:586
#5  0x00000000007b165a in read_reschedule (nh=0x475fea0, vc=0x2b4f7456b3e0) at UnixNetVConnection.cc:44
#6  0x00000000007b2009 in read_from_net (nh=0x475fea0, vc=0x2b4f7456b3e0, thread=0x475c000) at UnixNetVConnection.cc:253
#7  0x00000000007b4895 in UnixNetVConnection::net_read_io (this=0x2b4f7456b3e0, nh=0x475fea0, lthread=0x475c000)
    at UnixNetVConnection.cc:961
#8  0x00000000007a92e5 in NetHandler::waitForActivity (this=0x475fea0, timeout=60000000) at UnixNet.cc:497
#9  0x00000000007d5cf1 in EThread::execute_regular (this=0x475c000) at UnixEThread.cc:248
#10 0x00000000007d5df0 in EThread::execute (this=0x475c000) at UnixEThread.cc:278
#11 0x00000000007d4cbb in spawn_thread_internal (a=0x39676e0) at Thread.cc:84
#12 0x00002b4f68cfbea5 in start_thread () from /lib64/libpthread.so.0
#13 0x00002b4f69a318cd in clone () from /lib64/libc.so.6

So not quite the same stack trace...