test_Alternate_S_to_L_remove_S and test_Alternate_S_to_L_remove_L core dump

[traeak]

Found this while performing internal builds for ats92. Tracked this to PR #8718 which switched some tests from EXPENSIVE to normal. Verified the dumps on master.

This only happened when doing a fresh build with make -j check. Running the tests individually worked fine (no mem fault errors).

../../build/_aux/test-driver: line 95:  5471 Aborted                 (core dumped) "$@" > $log_file 2>&1
FAIL: test_Alternate_S_to_L_remove_S
PASS: test_Alternate_L_to_S_remove_L
../../build/_aux/test-driver: line 95:  5474 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: test_Alternate_S_to_L_remove_L

The best I have for this is from backtrace (no symbols):

1 lt-test_Alternate_S_to_L_remove_L error on 1 host
Callstack
object_key_get CacheAltTest_S_to_L_remove_L::delete_earliest_dir(CacheVC*) CacheAltTest_S_to_L_remove_L::handle_cache_event(int, CacheTestBase*) CacheReadTest::read_event(int, void*) CacheVC::openReadMain(int, Event*) handleEvent AIOCallbackInternal::io_complete(int, void*) EThread::process_event(Event*, int) EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*) EThread::execute_regular() execute EThread::execute() spawn_thread_internal

1 lt-test_Alternate_S_to_L_remove_S error on 1 host
Callstack
abort std::terminate() Catch::AssertionHandler::complete() [clone .part.0] Catch::AssertionHandler::complete() [clone .cold] CacheAltReadAgain::handle_cache_event(int, CacheTestBase*) CacheReadTest::read_event(int, void*) CacheVC::openReadStartEarliest(int, Event*) handleEvent AIOCallbackInternal::io_complete(int, void*) EThread::process_event(Event*, int) EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*) EThread::execute_regular() execute EThread::execute() spawn_thread_internal

[traeak]

Better stack trace

Core was generated by `/home/bolsen149/master/trafficserver/iocore/cache/.libs/lt-test_Alternate_S_to_'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000580394 in object_key_get (hash=<optimized out>, 
    this=<optimized out>) at ./test/test_Alternate_S_to_L_remove_L.cc:228
228         REQUIRE(dir_probe(&key, vc->vol, &dir, &last_collision) != 0);
Missing separate debuginfos, use: debuginfo-install glibc-2.17-325.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 openssl11-libs-1.1.1k-2.el7.x86_64 pcre-8.32-17.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-19.el7_9.x86_64
(gdb) where
#0  0x0000000000580394 in object_key_get (hash=<optimized out>, 
    this=<optimized out>) at ./test/test_Alternate_S_to_L_remove_L.cc:228
#1  CacheAltTest_S_to_L_remove_L::delete_earliest_dir (
    this=this@entry=0x1539b60, vc=0x157f880)
    at ./test/test_Alternate_S_to_L_remove_L.cc:227
#2  0x0000000000580784 in CacheAltTest_S_to_L_remove_L::handle_cache_event (
    this=0x1539b60, event=<optimized out>, base=0x1539bc0)
    at ./test/test_Alternate_S_to_L_remove_L.cc:199
#3  0x000000000052df3a in process_event (event=<optimized out>, this=0x1539bc0)
    at ./test/main.h:131
#4  CacheReadTest::read_event(int, void*) () at ./test/main.cc:271
#5  0x00000000005bb159 in handleEvent (data=0x157fae8, event=102, 
    this=<optimized out>)
    at /home/bolsen149/master/trafficserver/iocore/eventsystem/I_Continuation.h:219
#6  handleEvent (data=0x157fae8, event=102, this=<optimized out>)
    at /home/bolsen149/master/trafficserver/iocore/eventsystem/I_Continuation.h:215
#7  calluser (event=102, this=<optimized out>) at P_CacheInternal.h:636
#8  CacheVC::openReadMain(int, Event*) () at CacheRead.cc:780
#9  0x0000000000597161 in handleEvent (data=0x157fa10, event=3900, 
    this=0x157f880)
    at /home/bolsen149/master/trafficserver/iocore/eventsystem/I_Continuation.h:219
#10 handleEvent (data=0x157fa10, event=3900, this=0x157f880)
    at /home/bolsen149/master/trafficserver/iocore/eventsystem/I_Continuation.h:215
#11 AIOCallbackInternal::io_complete (this=0x157fa10, event=<optimized out>, 
    data=<optimized out>)
    at /home/bolsen149/master/trafficserver/iocore/aio/P_AIO.h:122
#12 0x000000000086da23 in handleEvent (data=0x152ea60, event=1, this=0x157fa10)
    at I_Continuation.h:219
#13 handleEvent (data=0x152ea60, event=1, this=0x157fa10)
    at I_Continuation.h:215
#14 EThread::process_event(Event*, int) () at UnixEThread.cc:164
#15 0x000000000086e00e in EThread::process_queue (
    this=this@entry=0x7fb2686f3010, 
    NegativeQueue=NegativeQueue@entry=0x7fb2686f0480, 
    ev_count=ev_count@entry=0x7fb2686f047c, 
    nq_count=nq_count@entry=0x7fb2686f0478) at UnixEThread.cc:199
#16 0x000000000086e44b in EThread::execute_regular (
    this=this@entry=0x7fb2686f3010) at UnixEThread.cc:255
#17 0x000000000086eac2 in execute (this=0x7fb2686f3010) at UnixEThread.cc:360
#18 EThread::execute (this=0x7fb2686f3010) at UnixEThread.cc:338
#19 0x000000000086c8a2 in spawn_thread_internal (a=0x152fa80) at Thread.cc:91
#20 0x00007fb26a0faea5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fb2693ffb0d in clone () from /lib64/libc.so.6

[bryancall]

@cmcfarlen is going to see if he can verify it with gcc11 and Centos 7