help: segfault in boost::asio::detail::epoll_reactor::start_op while upload_from_stream in multiple threads

[yxiang92128]

HI @Jinming-Hu

We hit a segfault while performing performance test with multiple threads (8 sessions) uploading 16 blobs simultaneously to the same bucket.
Again our environment is:
OS: SLES12SP3
Azure Storage C++ SDK: 7.0.0
REST C++ SDK: 2.10.14
libboost: 1.58

I wonder if you've seen the following trace before and any possible hint to work around it. Looks like the "descriptor_data" is 0x0 in start_op.

Again here is the call I am making to upload the 16MB buffer where iface is an object we pass in which contains the path and also the buffer we use to upload.

   // Construct a URI
    std::string fullname = std::string() + iface->uri_primary().to_string().c_str() + (iface->uri_primary().is_path_empty()?"":"/") + iface->m_bucket + "/" + obj_name;

    // Encodes the URI component according to RFC 3986 so uri constructor will not throw an exception
    auto encoded_filename = web::http::uri(web::http::uri::encode_uri(_XPLATSTR(fullname)));
    azure::storage::storage_uri blob_uri_new = azure::storage::storage_uri(encoded_filename);
    block_blob = azure::storage::cloud_block_blob(blob_uri_new, blob_client.credentials());

    azure::storage::blob_request_options reqOptions;


    reqOptions.set_parallelism_factor(1);

    reqOptions.set_maximum_execution_time(std::chrono::milliseconds(3000));
    reqOptions.set_server_timeout(std::chrono::seconds(600));

    // write operation would only target primary location no retry is required on secondary
    reqOptions.set_location_mode(azure::storage::location_mode::primary_only);


    azure::storage::operation_context context;
    ApplySSEAzure(sseinfo, context);

    block_blob.upload_from_stream((*iface->m_istream), olen, azure::storage::access_condition(), reqOptions, context);

    //block_blob.upload_from_stream((*iface->m_istream), azure::storage::access_condition(), reqOptions,


    // change it to a per object based shared ptr of input stream to avoid race condition
    iface->m_istream->close().wait();
  }
  catch (const azure::storage::storage_exception& e)
  {

Let me know if you find anything while I am also digging.

Many thanks,

Yang

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f3ef4ed0565 in boost::asio::detail::epoll_reactor::start_op (this=0x7f3ce009b6f0, op_type=1, descriptor=10, descriptor_data=@0x7f3d940404d0: 0x0, op=0x7f3c2401fdf0,
is_continuation=true, allow_speculative=true) at /usr/local/include/boost/asio/detail/impl/epoll_reactor.ipp:219
219 /usr/local/include/boost/asio/detail/impl/epoll_reactor.ipp: No such file or directory.
(gdb) bt
#0 0x00007f3ef4ed0565 in boost::asio::detail::epoll_reactor::start_op (this=0x7f3ce009b6f0, op_type=1, descriptor=10, descriptor_data=@0x7f3d940404d0: 0x0, op=0x7f3c2401fdf0,
is_continuation=true, allow_speculative=true) at /usr/local/include/boost/asio/detail/impl/epoll_reactor.ipp:219
#1 0x00007f3ef4ed6a18 in boost::asio::detail::reactive_socket_service_base::start_op (this=0x7f3ce00bbf88, impl=..., op_type=1, op=0x7f3c2401fdf0, is_continuation=true,
is_non_blocking=true, noop=false) at /usr/local/include/boost/asio/detail/impl/reactive_socket_service_base.ipp:214
#2 0x00007f3ef4ff92a2 in boost::asio::detail::reactive_socket_service_base::async_send<boost::asio::mutable_buffers_1, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > > (this=0x7f3ce00bbf88, impl=..., buffers=..., flags=0, handler=...) at /usr/local/include/boost/asio/detail/reactive_socket_service_base.hpp:216
#3 0x00007f3ef4ff7210 in boost::asio::stream_socket_serviceboost::asio::ip::tcp::async_send<boost::asio::mutable_buffers_1, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > >(boost::asio::detail::reactive_socket_serviceboost::asio::ip::tcp::implementation_type&, boost::asio::mutable_buffers_1 const&, int, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >&&) (this=0x7f3ce00bbf60, impl=..., buffers=..., flags=0,
handler=<unknown type in /opt/teradata/tdobject/lib/libcpprest.so.2.10, CU 0xb62f0c, DIE 0xc5f8a4>) at /usr/local/include/boost/asio/stream_socket_service.hpp:330
#4 0x00007f3ef4ff5102 in boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >::async_write_some<boost::asio::mutable_buffers_1, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > >(boost::asio::mutable_buffers_1 const&, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >&&) (this=0x7f3d940404c8, buffers=..., handler=<unknown type in /opt/teradata/tdobject/lib/libcpprest.so.2.10, CU 0xb62f0c, DIE 0xc5c49b>)
at /usr/local/include/boost/asio/basic_stream_socket.hpp:732
#5 0x00007f3ef4ff273c in boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_strea---Type to continue, or q to quit---
mbuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >::operator() (this=0x7f3e893fc300, ec=..., bytes_transferred=0, start=1)
at /usr/local/include/boost/asio/impl/write.hpp:258
#6 0x00007f3ef4fefa18 in boost::asio::async_write<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >(boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&, boost::asio::mutable_buffers_1 const&, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > >&&) (s=..., buffers=...,
handler=<unknown type in /opt/teradata/tdobject/lib/libcpprest.so.2.10, CU 0xb62f0c, DIE 0xc53a5a>) at /usr/local/include/boost/asio/impl/write.hpp:621
#7 0x00007f3ef4fec1eb in boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > >::operator() (this=0x7f3e893fc550, ec=..., bytes_transferred=0, start=1)
at /usr/local/include/boost/asio/ssl/detail/io.hpp:189
#8 0x00007f3ef4fe8853 in boost::asio::ssl::detail::async_io<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > (next_layer=..., core=..., op=..., handler=...) at /usr/local/include/boost/asio/ssl/detail/io.hpp:337
#9 0x00007f3ef4fe494a in boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>::async_write_some<boost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > >(boost::asio::const_buffers_1 const&, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > >&&) (this=0x7f3d940f3e20, buffers=...,
handler=<unknown type in /opt/teradata/tdobject/lib/libcpprest.so.2.10, CU 0xb62f0c, DIE 0xc3fe4e>) at /usr/local/include/boost/asio/ssl/stream.hpp:646
#10 0x00007f3ef4fe0f06 in boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > >::operator() (this=0x7f3e893fcba8, ec=..., bytes_transferred=16384, start=0) at /usr/local/include/boost/asio/impl/write.hpp:334
#11 0x00007f3ef4fefabc in boost::asio::ssl::detail::write_opboost::asio::const_buffers_1::call_handler<boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > (this=0x7f3e893fcb78, handler=..., ec=...,
---Type to continue, or q to quit---
bytes_transferred=@0x7f3e893fc860: 16384) at /usr/local/include/boost/asio/ssl/detail/write_op.hpp:59
#12 0x00007f3ef4fec46f in boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > >::operator() (this=0x7f3e893fcb68, ec=..., bytes_transferred=16413, start=0)
at /usr/local/include/boost/asio/ssl/detail/io.hpp:266
#13 0x00007f3ef4ff27f0 in boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >::operator() (this=0x7f3e893fcb40, ec=..., bytes_transferred=16413,
start=0) at /usr/local/include/boost/asio/impl/write.hpp:269
#14 0x00007f3ef4ffeffb in boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>::operator() (this=0x7f3e893fcb40) at /usr/local/include/boost/asio/detail/bind_handler.hpp:127
#15 0x00007f3ef4ffef8b in boost::asio::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long> > (function=...) at /usr/local/include/boost/asio/handler_invoke_hook.hpp:69
#16 0x00007f3ef4ffee45 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > (function=..., context=...)
at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#17 0x00007f3ef4ffeccb in boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boo---Type to continue, or q to quit---
st::system::error_code, unsigned long>, std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > (function=..., this_handler=0x7f3e893fcbd0)
at /usr/local/include/boost/asio/impl/write.hpp:699
#18 0x00007f3ef4ffeb67 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > (
function=..., context=...) at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#19 0x00007f3ef4ffe915 in boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > (function=..., this_handler=0x7f3e893fcba8) at /usr/local/include/boost/asio/impl/write.hpp:565
#20 0x00007f3ef4ffe559 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > (function=..., context=...) at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#21 0x00007f3ef4ffe147 in boost::asio::ssl::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > (function=..., this_handler=0x7f3e893fcb68) at /usr/local/include/boost/asio/ssl/detail/io.hpp:320
---Type to continue, or q to quit---
#22 0x00007f3ef4ffdd2a in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > (function=..., context=...)
at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#23 0x00007f3ef4ffd805 in boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > (function=..., this_handler=0x7f3e893fcb40)
at /usr/local/include/boost/asio/impl/write.hpp:565
#24 0x00007f3ef4ffd10d in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > >, boost::system::error_code, unsigned long>, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > > > > > > (function=..., context=...)
at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#25 0x00007f3ef4ffc371 in boost::asio::detail::reactive_socket_send_op<boost::asio::mutable_buffers_1, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, boost::asio::ssl::detail::write_opboost::asio::const_buffers_1, boost::asio::detail::write_op<boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >&>, boost::asio::const_buffers_1, boost::asio::detail::transfer_all_t, boost::asio::detail::write_streambuf_handler<std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::de---Type to continue, or q to quit---
tails::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> (*)()> > > > > > >::do_complete (owner=0x7f3ce00613e0, base=0x7f3c2401fdf0) at /usr/local/include/boost/asio/detail/reactive_socket_send_op.hpp:107
#26 0x00007f3ef4ecf550 in boost::asio::detail::task_io_service_operation::complete (this=0x7f3c2401fdf0, owner=..., ec=..., bytes_transferred=0)
at /usr/local/include/boost/asio/detail/task_io_service_operation.hpp:38
#27 0x00007f3ef4ed2058 in boost::asio::detail::task_io_service::do_run_one (this=0x7f3ce00613e0, lock=..., this_thread=..., ec=...)
at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:372
#28 0x00007f3ef4ed1a8c in boost::asio::detail::task_io_service::run (this=0x7f3ce00613e0, ec=...) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:149
#29 0x00007f3ef4ed2345 in boost::asio::io_service::run (this=0x7f3ef53fd248 <(anonymous namespace)::initialize_shared_threadpool(unsigned long)::uninit_threadpool+8>)
at /usr/local/include/boost/asio/impl/io_service.ipp:59
#30 0x00007f3ef4fc3db9 in (anonymous namespace)::threadpool_impl::thread_start (
arg=0x7f3ef53fd240 <(anonymous namespace)::initialize_shared_threadpool(unsigned long)::uninit_threadpool>)
at /home/jenkins/cpprestsdk-2.10.14/Release/src/pplx/threadpool.cpp:79
#31 0x00007f3ef4fc3ced in (anonymous namespace)::threadpool_impl::<lambda()>::operator()(void) const (__closure=0x7f3ce0062bd8)
at /home/jenkins/cpprestsdk-2.10.14/Release/src/pplx/threadpool.cpp:64
#32 0x00007f3ef4fc47a2 in boost::asio::detail::posix_thread::func<(anonymous namespace)::threadpool_impl::add_thread()::<lambda()> >::run(void) (this=0x7f3ce0062bd0)
at /usr/local/include/boost/asio/detail/posix_thread.hpp:82
#33 0x00007f3ef4ed72c9 in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x7f3ce0062bd0) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:64
#34 0x00007f41f94106b4 in start_thread () from /lib64/libpthread.so.0
#35 0x00007f41f2ed22dd in clone () from /lib64/libc.so.6
(gdb)
(gdb) print descriptor_data
$1 = (boost::asio::detail::epoll_reactor::per_descriptor_data &) @0x7f3d940404d0: 0x0
(gdb)

[katmsft]

It seems to be a known issue for boost, the linked issue has a similar trace: arvidn/libtorrent#3084
There isn't a good update in the origin post for the root cause: https://svn.boost.org/trac10/ticket/7611#no2
I believe this is a CPPRest SDK issue, there is nothing that can be done from this SDK's perspective. This conclusion is drawn due to the linked issue being resolved by disabling the network thread pool, which is purely controlled by CPPRest.

[yxiang92128]

Thanks for the reply.
Are you saying there is a way to disable the network thread pool in CPPREST SDK? Shall I reach out to their github for help?
Any hint would be appreciated, our feature release is highly dependent on this to work.
Thanks again,

Yang

[katmsft]

Are you saying there is a way to disable the network thread pool in CPPREST SDK?

No, I don't think there is an easy way but it would require them to confirm or provide workarounds.