Crash in Http1ClientSession::free (triggered via authproxy plugin)

[zwoop]

Getting this crash using latest v9.0.x:

#0  0x00007f3c2ba2a199 in waitpid () from /lib64/libpthread.so.0
#1  0x000055e59c34c254 in crash_logger_invoke (signo=11, info=0x7f3c2270acb0, ctx=0x7f3c2270ab80) at traffic_server/Crash.cc:168
#2  <signal handler called>
#3  0x0000000000000000 in ?? ()
#4  0x000055e59c39af72 in Http1ClientSession::free (this=0x7f3aeb6cd820) at Http1ClientSession.cc:115
#5  0x000055e59c5480f4 in handle_api_return (event=60000, this=<optimized out>) at ProxySession.cc:168
#6  ProxySession::do_api_callout (this=0x7f3aeb6cd820, id=<optimized out>) at ProxySession.cc:145
#7  0x000055e59c39ae99 in Http1ClientSession::destroy (this=0x7f3aeb6cd820) at Http1ClientSession.cc:79
#8  0x000055e59c3b254f in HttpSM::kill_this (this=0x7f3b10435800) at HttpSM.cc:6946
#9  0x000055e59c3b8c52 in HttpSM::state_api_callback (this=0x7f3b10435800, event=60000, data=0x0) at HttpSM.cc:1351
#10 0x000055e59c365f64 in TSHttpTxnReenable (txnp=0x7f3b10435800, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6090
#11 0x00007f3bfeb66878 in StateAuthorized (auth=0x7f3c21f20060) at authproxy/authproxy.cc:615
#12 0x00007f3bfeb65d05 in AuthRequestContext::dispatch (cont=<optimized out>, event=TS_EVENT_IMMEDIATE, edata=0x7f004e161f10) at authproxy/authproxy.cc:218
#13 0x000055e59c35810e in INKContInternal::handle_event (this=0x7f3c0cfdd3a0, event=100, edata=0x7f004e161f10) at traffic_server/InkAPI.cc:1095
#14 0x000055e59c5433f0 in handleEvent (event=100, this=<optimized out>, data=<optimized out>)
    at /usr/src/debug/bazinga-trafficserver-9.0.0/iocore/eventsystem/I_Continuation.h:190
#15 PluginVC::process_read_side (this=0x7f004e161d40, other_side_call=true) at PluginVC.cc:659
#16 0x000055e59c543d7e in PluginVC::process_write_side (this=0x7f004e162008, other_side_call=<optimized out>) at PluginVC.cc:569
#17 0x000055e59c5424b4 in PluginVC::main_handler (this=0x7f004e162008, event=<optimized out>, data=<optimized out>) at PluginVC.cc:225
#18 0x000055e59c5af325 in handleEvent (this=<optimized out>, event=<optimized out>, data=<optimized out>) at ./I_Continuation.h:190
#19 EThread::process_event (this=0x7f3c2660d000, e=0x7f3ba24d8e20, calling_code=1) at UnixEThread.cc:137
#20 0x000055e59c5af5db in EThread::process_queue (this=0x7f3c2660d000, NegativeQueue=0x7f3c2270bde0, ev_count=0x7f3c2270be14, nq_count=0x7f3c2270bdfc)
    at UnixEThread.cc:176
#21 0x000055e59c5afa7b in EThread::execute_regular (this=0x7f3c2660d000) at UnixEThread.cc:236
#22 0x000055e59c5aff64 in EThread::execute (this=0x7f3c2660d000) at UnixEThread.cc:345
#23 0x000055e59c5ae97b in spawn_thread_internal (a=0x7f3c2a0bd230) at Thread.cc:92
#24 0x00007f3c2ba22e65 in start_thread () from /lib64/libpthread.so.0
#25 0x00007f3c2ae4088d in clone () from /lib64/libc.so.6

Full bt:

(gdb) bt full
#0  0x00007f3c2ba2a199 in waitpid () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x000055e59c34c254 in crash_logger_invoke (signo=11, info=0x7f3c2270acb0, ctx=0x7f3c2270ab80) at traffic_server/Crash.cc:168
        status = 0
#2  <signal handler called>
No symbol table info available.
#3  0x0000000000000000 in ?? ()
No symbol table info available.
#4  0x000055e59c39af72 in Http1ClientSession::free (this=0x7f3aeb6cd820) at Http1ClientSession.cc:115
No locals.
#5  0x000055e59c5480f4 in handle_api_return (event=60000, this=<optimized out>) at ProxySession.cc:168
        hookid = TS_HTTP_SSN_CLOSE_HOOK
#6  ProxySession::do_api_callout (this=0x7f3aeb6cd820, id=<optimized out>) at ProxySession.cc:145
No locals.
#7  0x000055e59c39ae99 in Http1ClientSession::destroy (this=0x7f3aeb6cd820) at Http1ClientSession.cc:79
No locals.
#8  0x000055e59c3b254f in HttpSM::kill_this (this=0x7f3b10435800) at HttpSM.cc:6946
No locals.
#9  0x000055e59c3b8c52 in HttpSM::state_api_callback (this=0x7f3b10435800, event=60000, data=0x0) at HttpSM.cc:1351
No locals.
#10 0x000055e59c365f64 in TSHttpTxnReenable (txnp=0x7f3b10435800, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6090
        trylock = {
          m = {
            m_ptr = 0x7f3b4c1bd210
          },
          lock_acquired = true
        }
        sm = 0x7f3b10435800
        eth = 0x7f3c2660d000
        cb = <optimized out>
#11 0x00007f3bfeb66878 in StateAuthorized (auth=0x7f3c21f20060) at authproxy/authproxy.cc:615
        options = 0x7f3ba4a373e0
#12 0x00007f3bfeb65d05 in AuthRequestContext::dispatch (cont=<optimized out>, event=TS_EVENT_IMMEDIATE, edata=0x7f004e161f10) at authproxy/authproxy.cc:218
        auth = 0x7f3c21f20060
        s = <optimized out>
#13 0x000055e59c35810e in INKContInternal::handle_event (this=0x7f3c0cfdd3a0, event=100, edata=0x7f004e161f10) at traffic_server/InkAPI.cc:1095
        retval = <optimized out>
#14 0x000055e59c5433f0 in handleEvent (event=100, this=<optimized out>, data=<optimized out>) at /usr/src/debug/bazinga-trafficserver-9.0.0/iocore/eventsystem/I_Continuation.h:190
No locals.
#15 PluginVC::process_read_side (this=0x7f004e161d40, other_side_call=true) at PluginVC.cc:659
        core_reader = <optimized out>
        ntodo = <optimized out>
        bytes_avail = <optimized out>
        act_on = 1247
        output_buffer = <optimized out>
        water_mark = <optimized out>
        buf_space = <optimized out>
        added = <optimized out>
#16 0x000055e59c543d7e in PluginVC::process_write_side (this=0x7f004e162008, other_side_call=<optimized out>) at PluginVC.cc:569
        my_ethread = <optimized out>
        lock = <optimized out>
        core_buffer = <optimized out>
        ntodo = <optimized out>
        reader = <optimized out>
        bytes_avail = <optimized out>
        act_on = <optimized out>
        buf_space = <optimized out>
        added = <optimized out>
#17 0x000055e59c5424b4 in PluginVC::main_handler (this=0x7f004e162008, event=<optimized out>, data=<optimized out>) at PluginVC.cc:225
        call_event = <optimized out>
        my_ethread = 0x7f3c2660d000
        read_mutex_held = true
        write_mutex_held = <optimized out>
#18 0x000055e59c5af325 in handleEvent (this=<optimized out>, event=<optimized out>, data=<optimized out>) at ./I_Continuation.h:190
No locals.
#19 EThread::process_event (this=0x7f3c2660d000, e=0x7f3ba24d8e20, calling_code=1) at UnixEThread.cc:137
        c_temp = <optimized out>
        lock = {
          m = {
            m_ptr = 0x7f3b8cb2d9e0
          },
          lock_acquired = true
        }
#20 0x000055e59c5af5db in EThread::process_queue (this=0x7f3c2660d000, NegativeQueue=0x7f3c2270bde0, ev_count=0x7f3c2270be14, nq_count=0x7f3c2270bdfc) at UnixEThread.cc:176
No locals.
#21 0x000055e59c5afa7b in EThread::execute_regular (this=0x7f3c2660d000) at UnixEThread.cc:236
        sleep_time = <optimized out>
        done_one = <optimized out>
        METRIC_INIT = {
          _loop_time = {
            _start = 0,
            _min = 9223372036854775807,
            _max = 0
          },
          _events = {
            _min = 2147483647,
            _max = 0,
            _total = 0
          },
          _count = 0,
          _wait = 0
        }
        prev_metric = 0x7f3c26717f90
        ev_count = 1
        nq_count = 0
        loop_start_time = <optimized out>
        next_time = <optimized out>
        loop_finish_time = <optimized out>
        delta = <optimized out>
#22 0x000055e59c5aff64 in EThread::execute (this=0x7f3c2660d000) at UnixEThread.cc:345
No locals.
#23 0x000055e59c5ae97b in spawn_thread_internal (a=0x7f3c2a0bd230) at Thread.cc:92
        p = 0x7f3c2a0bd230
#24 0x00007f3c2ba22e65 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#25 0x00007f3c2ae4088d in clone () from /lib64/libc.so.6
No symbol table info available.

HttpSM History (frame 8):

(gdb) p history
$2 = {
  history = {{
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5c9efd "state_read_client_request_header",
        line = 613
      },
      event = 100,
      reentrancy = 2
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca39a "state_api_callback",
        line = 1337
      },
      event = 60000,
      reentrancy = 3
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca3b6 "state_api_callout",
        line = 1377
      },
      event = 60000,
      reentrancy = 3
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca39a "state_api_callback",
        line = 1337
      },
      event = 60000,
      reentrancy = 4
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca3b6 "state_api_callout",
        line = 1377
      },
      event = 60000,
      reentrancy = 4
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca39a "state_api_callback",
        line = 1337
      },
      event = 60000,
      reentrancy = 5
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca3b6 "state_api_callout",
        line = 1377
      },
      event = 60000,
      reentrancy = 5
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca39a "state_api_callback",
        line = 1337
      },
      event = 60000,
      reentrancy = 6
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca3b6 "state_api_callout",
        line = 1377
      },
      event = 60000,
      reentrancy = 6
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca169 "state_watch_for_client_abort",
        line = 880
      },
      event = 105,
      reentrancy = 1
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca39a "state_api_callback",
        line = 1337
      },
      event = 60000,
      reentrancy = 1
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca3b6 "state_api_callout",
        line = 1377
      },
      event = 60000,
      reentrancy = 1
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca39a "state_api_callback",
        line = 1337
      },
      event = 60000,
      reentrancy = 2
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca3b6 "state_api_callout",
        line = 1377
      },
      event = 60000,
      reentrancy = 2
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca39a "state_api_callback",
        line = 1337
      },
      event = 60000,
      reentrancy = 3
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca3b6 "state_api_callout",
        line = 1377
      },
      event = 60000,
      reentrancy = 3
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca39a "state_api_callback",
        line = 1337
      },
      event = 60000,
      reentrancy = 4
    }, {
      location = {
        file = 0x55e59c5c9e88 "HttpSM.cc",
        func = 0x55e59c5ca3b6 "state_api_callout",
        line = 1377
      },
      event = 60000,
      reentrancy = 4
    }, {
      location = {
        file = 0x0,
        func = 0x0,
        line = 0
      },
      event = 0,
      reentrancy = 0
    } <repeats 47 times>},
  history_pos = 18
}

[shinrich]

We are seeing a similar crash at a low rate (4 times in past month), but in our case, we are triggering from inactivity timeout instead of a read ready. In our core, the client_vc pointer in Http1ClientSession is not null, but the data pointed at is no longer a netvc. Only seeing this on the Http1 path.

(gdb) bt
#0  0x000000000051ed73 in Http1ClientSession::do_io_write (this=<optimized out>, c=0x0, nbytes=0, buf=0x0, owner=false)
    at ../../../../../../_vcs/trafficserver9/proxy/http/Http1ClientSession.cc:220
#1  0x000000000051f012 in Http1ClientSession::free (this=0x2af853a05300)
    at ../../../../../../_vcs/trafficserver9/proxy/http/Http1ClientSession.cc:115
#2  0x00000000006d1bd1 in ProxySession::handle_api_return (this=<optimized out>, event=<optimized out>)
    at ../../../../../_vcs/trafficserver9/proxy/ProxySession.cc:171
#3  0x00000000006d220d in ProxySession::do_api_callout (this=this@entry=0x2af853a05300, id=id@entry=TS_HTTP_SSN_CLOSE_HOOK)
    at ../../../../../_vcs/trafficserver9/proxy/ProxySession.cc:148
#4  0x0000000000520fcd in destroy (this=0x2af853a05300) at ../../../../../../_vcs/trafficserver9/proxy/http/Http1ClientSession.cc:79
#5  Http1ClientSession::destroy (this=0x2af853a05300) at ../../../../../../_vcs/trafficserver9/proxy/http/Http1ClientSession.cc:68
#6  0x000000000054a911 in HttpSM::kill_this() () at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:6951
#7  0x000000000054ad70 in main_handler (data=0x2aebf37f5f40, event=2301, this=0x2aebf37f5000)
    at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2625
#8  HttpSM::main_handler (this=0x2aebf37f5000, event=2301, data=0x2aebf37f5f40)
    at ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2582
#9  0x000000000058fd29 in handleEvent (data=0x2aebf37f5f40, event=2301, this=0x2aebf37f5000)
    at /sd/workspace/src/git.ouroath.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#10 handleEvent (data=0x2aebf37f5f40, event=2301, this=0x2aebf37f5000)
    at /sd/workspace/src/git.ouroath.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#11 HttpTunnel::main_handler (this=0x2aebf37f5f40, event=<optimized out>, data=<optimized out>)
    at ../../../../../../_vcs/trafficserver9/proxy/http/HttpTunnel.cc:1625
#12 0x0000000000716705 in handleEvent (data=0x2af853658ed0, event=105, this=0x2aebf37f5f40)
    at /sd/workspace/src/git.ouroath.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#13 handleEvent (data=0x2af853658ed0, event=105, this=0x2aebf37f5f40)
    at /sd/workspace/src/git.ouroath.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#14 read_signal_and_update (event=105, vc=0x2af853658d00) at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:83
#15 0x000000000071abec in UnixNetVConnection::mainEvent(int, Event*) ()
    at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:1152
#16 0x000000000070e35d in handleEvent (data=0x2add9dc00060, event=105, this=0x2af853658d00)
    at /sd/workspace/src/git.ouroath.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#17 handleEvent (data=0x2add9dc00060, event=105, this=0x2af853658d00)
    at /sd/workspace/src/git.ouroath.com/Edge/build/_build/build_release_posix-x86_64_gcc_8/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#18 InactivityCop::check_inactivity (this=0x2add9dccb000, event=<optimized out>, e=<optimized out>)
    at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:81
#19 0x00000000007590cb in handleEvent (data=0x2add9dc00060, event=2, this=0x2add9dccb000)
    at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#20 handleEvent (data=0x2add9dc00060, event=2, this=0x2add9dccb000)
    at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#21 EThread::process_event(Event*, int) () at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:132
#22 0x0000000000759f84 in EThread::execute_regular (this=this@entry=0x2add99f5be80)
    at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:241
#23 0x000000000075a396 in execute (this=0x2add99f5be80) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
#24 EThread::execute (this=0x2add99f5be80) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
#25 0x0000000000758619 in spawn_thread_internal (a=0x2add97db5740)
    at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
---Type <return> to continue, or q <return> to quit--- 
#26 0x00002add960c2dd5 in start_thread () from /lib64/libpthread.so.0
#27 0x00002add96df7ead in clone () from /lib64/libc.so.6

Was seeing this with our older 9.0.x branch and yesterday with our latest, greatest sync. (appears to have commits from 3-10 and earlier).

[shinrich]

Our issue anyway seems related to PR #6469 but that fix is already on 9.0.x, so it must be an additional case.

[shinrich]

@bneradt got the following asan stack which seems to be floating around the same area. In this case our do_io_close() in fact freed the Session object, but we are accessing the client_vc on the way out. At this point I think the stale client_vc pointer issue is due to calls to do_io_read/write that clear the continuation. If we are unlucky and the vc closes itself at this time there is no way for the netvc layer to signal back to the SM.

==60923==ERROR: AddressSanitizer: heap-use-after-free on address 0x6190000dffb0 at pc 0x00000077e141 bp 0x2adc45476cd0 sp 0x2adc45476cc0
READ of size 8 at 0x6190000dffb0 thread T16 ([ET_NET 14])
    #0 0x77e140 in Http1ClientSession::state_keep_alive(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/Http1ClientSession.cc:399
    #1 0xec4594 in Continuation::handleEvent(int, void*) /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
    #2 0xec4594 in Continuation::handleEvent(int, void*) /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
    #3 0xec4594 in read_signal_and_update ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:83
    #4 0xec4594 in read_signal_done ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:144
    #5 0xedad60 in read_from_net ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:279
    #6 0xe995c6 in NetHandler::process_ready_list() ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:401
    #7 0xe9a4a6 in NetHandler::waitForActivity(long) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:536
    #8 0xfc46be in EThread::execute_regular() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:271
    #9 0xfc4f15 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
    #10 0xfc4f15 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
    #11 0xfbe9fa in spawn_thread_internal ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
    #12 0x2adc3b82fdd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)
    #13 0x2adc3c76d02c in __clone (/lib64/libc.so.6+0xfe02c)

0x6190000dffb0 is located 816 bytes inside of 1104-byte region [0x6190000dfc80,0x6190000e00d0)
freed by thread T16 ([ET_NET 14]) here:
    #0 0x61bb10 in free (/opt/oath/trafficserver/9.0/bin/traffic_server+0x61bb10)
    #1 0x2adc39e6689c in ink_freelist_free ../../../../../../_vcs/trafficserver9/src/tscore/ink_queue.cc:277
    #2 0xdc657a in ProxySession::handle_api_return(int) ../../../../../_vcs/trafficserver9/proxy/ProxySession.cc:171
    #3 0xdc6a73 in ProxySession::state_api_callout(int, void*) ../../../../../_vcs/trafficserver9/proxy/ProxySession.cc:126
    #4 0x6df416 in TSHttpSsnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:4908
    #5 0x2adc47ce96cb in global_ssn_handler ../../../../../_vcs/trafficserver9/plugins/experimental/traffic_dump/traffic_dump.cc:607
    #6 0x69cab8 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1102
    #7 0x6d5e62 in Continuation::handleEvent(int, void*) /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
    #8 0x6d5e62 in Continuation::handleEvent(int, void*) /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
    #9 0x6d5e62 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1339
    #10 0xdc6d69 in ProxySession::state_api_callout(int, void*) ../../../../../_vcs/trafficserver9/proxy/ProxySession.cc:117
    #11 0x77d00c in Http1ClientSession::destroy() ../../../../../../_vcs/trafficserver9/proxy/http/Http1ClientSession.cc:79
    #12 0x77adec in Http1ClientSession::do_io_close(int) ../../../../../../_vcs/trafficserver9/proxy/http/Http1ClientSession.cc:295
    #13 0x77dd2e in Http1ClientSession::state_keep_alive(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/Http1ClientSession.cc:398
    #14 0xec4594 in Continuation::handleEvent(int, void*) /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
    #15 0xec4594 in Continuation::handleEvent(int, void*) /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
    #16 0xec4594 in read_signal_and_update ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:83
    #17 0xec4594 in read_signal_done ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:144
    #18 0xedad60 in read_from_net ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:279
    #19 0xe995c6 in NetHandler::process_ready_list() ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:401
    #20 0xe9a4a6 in NetHandler::waitForActivity(long) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:536
    #21 0xfc46be in EThread::execute_regular() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:271
    #22 0xfc4f15 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
    #23 0xfc4f15 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
    #24 0xfbe9fa in spawn_thread_internal ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
    #25 0x2adc3b82fdd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)

previously allocated by thread T16 ([ET_NET 14]) here:
    #0 0x61ccf0 in posix_memalign (/opt/oath/trafficserver/9.0/bin/traffic_server+0x61ccf0)
    #1 0x2adc39e6478f in ats_memalign ../../../../../../_vcs/trafficserver9/src/tscore/ink_memory.cc:102
    #2 0x2adc39e667fe in ink_freelist_new ../../../../../../_vcs/trafficserver9/src/tscore/ink_queue.cc:187
    #3 0x76433b in ClassAllocator<Http1ClientSession>::alloc() /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/include/tscore/Allocator.h:131
    #4 0x76433b in Http1ClientSession* thread_alloc_init<Http1ClientSession>(ClassAllocator<Http1ClientSession>&, ProxyAllocator&) /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_ProxyAllocator.h:73
    #5 0x76433b in HttpSessionAccept::accept(NetVConnection*, MIOBuffer*, IOBufferReader*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSessionAccept.cc:52
    #6 0xdc1d71 in ProtocolProbeTrampoline::ioCompletionEvent(int, void*) ../../../../../_vcs/trafficserver9/proxy/ProtocolProbeSessionAccept.cc:147
    #7 0xec3016 in Continuation::handleEvent(int, void*) /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
    #8 0xec3016 in Continuation::handleEvent(int, void*) /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
    #9 0xec3016 in read_signal_and_update ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:83
    #10 0xed9cba in read_from_net ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:309
    #11 0xe995c6 in NetHandler::process_ready_list() ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:401
    #12 0xe9a4a6 in NetHandler::waitForActivity(long) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:536
    #13 0xfc46be in EThread::execute_regular() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:271
    #14 0xfc4f15 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
    #15 0xfc4f15 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
    #16 0xfbe9fa in spawn_thread_internal ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
    #17 0x2adc3b82fdd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)

Thread T16 ([ET_NET 14]) created by T0 ([TS_MAIN]) here:
    #0 0x57f9af in pthread_create (/opt/oath/trafficserver/9.0/bin/traffic_server+0x57f9af)
    #1 0xfbfb8e in ink_thread_create /home/bneradt/repos/build_ts_9/_build/build_release_posix-x86_64/trafficserver9/build/../../../../_vcs/trafficserver9/include/tscore/ink_thread.h:159
    #2 0xfbfb8e in Thread::start(char const*, void*, unsigned long, std::function<void ()> const&) ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:109
    #3 0xfccbeb in EventProcessor::spawn_event_threads(int, int, unsigned long) ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEventProcessor.cc:392
    #4 0xfcd999 in EventProcessor::start(int, unsigned long) ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEventProcessor.cc:455
    #5 0x504fb1 in main ../../../../../_vcs/trafficserver9/src/traffic_server/traffic_server.cc:1970
    #6 0x2adc3c691494 in __libc_start_main (/lib64/libc.so.6+0x22494)

[ywkaras]

I'm going to try to write up a strategic proposal for these death order issues and send it to the dev mailing list.

[shinrich]

At this point PR #6889 has addressed our stale net_vc issues on 9.0.x.

[shinrich]

PR #6889 has landed in 9.0.x