SEGV on CacheDisk destructor

[masaori335]

[ 00 ] libpthread-2.17.so waitpid 
[ 01 ] traffic_server crash_logger_invoke(int, siginfo_t*, void*) ( Crash.cc:165 ) 
[ 02 ] libpthread-2.17.so 
[ 03 ] traffic_server CacheDisk::~CacheDisk() ( List.h:293 ) 
[ 04 ] traffic_server CacheDisk::~CacheDisk() ( CacheDisk.cc:112 ) 
[ 05 ] traffic_server CacheProcessor::diskInitialized() ( Cache.cc:767 ) 
[ 06 ] traffic_server CacheDisk::openStart(int, void*) ( CacheDisk.cc:213 ) 
[ 07 ] traffic_server AIOCallbackInternal::io_complete(int, void*) ( eventsystem/I_Continuation.h:160 ) 
[ 08 ] traffic_server EThread::process_event(Event*, int) ( I_Continuation.h:160 ) 
[ 09 ] traffic_server EThread::process_queue(Queue<Event, Event::Link_link>*, int*, int*) ( UnixEThread.cc:170 ) 
[ 10 ] traffic_server EThread::execute_regular() ( UnixEThread.cc:230 ) 
[ 11 ] traffic_server spawn_thread_internal(void*) ( Thread.cc:85 ) 
[ 12 ] libpthread-2.17.so start_thread 

Affected version: 8.1.2

[masaori335]

It looks like ATS found a bad disk and tried to delete it, but got SEGV on disk_vols[i]->dpb_queue.pop().

trafficserver/iocore/cache/Cache.cc

Lines 764 to 767 in cbb20d4

	// Check and remove bad disks from gdisks[]
	for (i = 0; i < gndisks; i++) {
	if (DISK_BAD(gdisks[i])) {
	delete gdisks[i];

trafficserver/iocore/cache/CacheDisk.cc

Line 117 in cbb20d4

while (disk_vols[i] && (q = (disk_vols[i]->dpb_queue.pop()))) {

[shinrich]

About 6 months ago, I spent many iterations trying to fix our crash on bad disk problem. What version are you running? I thought I had pushed those fixes to master.

[masaori335]

The version we got this crash is almost the same as the current head of the 8.1.x branch.

[shinrich]

Take a look at PR #7218. I thought I had to make multiple attempts here, but this is the only relevant looking PR I see on master. I'll review my internal work and make sure it all made to to ASF. It could be that I delayed making an ASF PR until we got it all under control.

[shinrich]

Take a look at PR #7218. I thought I had to make multiple attempts here, but this is the only relevant looking PR I see on master. I'll review my internal work and make sure it all made to to ASF. It could be that I delayed making an ASF PR until we got it all under control.

[shinrich]

Also take a look at PR #7397. That one just fixes things so traffic_manager gives up after failing to open the cache disks.