Httpclient calls with OpenSSL 1.0.1e crash

[y1015860449]

Crash core:

#0  0x00007fa48b4d3277 in raise () from /lib64/libc.so.6
#1  0x00007fa48b4d4968 in abort () from /lib64/libc.so.6
#2  0x00007fa48b515d37 in __libc_message () from /lib64/libc.so.6
#3  0x00007fa48b51c5e4 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fa48b521c39 in _int_realloc () from /lib64/libc.so.6
#5  0x00007fa48b522eb2 in realloc () from /lib64/libc.so.6
#6  0x00007fa48d2b5649 in CRYPTO_realloc () from ../lib/libcrypto.so.10
#7  0x00007fa48d330b11 in lh_insert () from ../lib/libcrypto.so.10
#8  0x00007fa48d3331e4 in int_thread_set_item () from ../lib/libcrypto.so.10
#9  0x00007fa48d333d6e in ERR_get_state () from ../lib/libcrypto.so.10
#10 0x00007fa48d333ef5 in ERR_clear_error () from ../lib/libcrypto.so.10
#11 0x00007fa48d6665b5 in ssl23_connect () from ../lib/libssl.so.10
#12 0x000000000049add6 in httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}::operator()(ssl_st*) const ()
#13 0x000000000049ed1b in bool httplib::detail::read_and_close_socket_ssl<httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#2}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*, httplib::Stream&, bool, bool&)#3}>(int, unsigned long, ssl_ctx_st*, std::mutex&, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#2}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*, httplib::Stream&, bool, bool&)#3}) ()
#14 0x000000000049afb0 in httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&) ()
#15 0x0000000000499c88 in httplib::Client::send(httplib::Request&, httplib::Response&) ()
#16 0x000000000049a57e in httplib::Client::Post(char const*, std::multimap<std::string, std::string, httplib::detail::ci, std::allocator<std::pair<std::string const, std::string> > > const&, std::string const&, char const*) ()

[yhirose]

@y1015860449, thanks for the report. Could you please give me more information about it to reproduce the problem on my machine for debugging?

[PixlRainbow]

Might want to have a catch somewhere in SSLClient or SSLServer to print the SSL error stack with ERR_print_errs()

[y1015860449]

@yhirose There are four crashes in two days. Use the thread pool to continuously send post requests.

[yhirose]

@y1015860449, sorry but I don't understand what you mean by 'the thread pool'. Could you explain clearly what you are actually doing by showing the small code snippet of your code? Also what OS and compiler are you using? Thanks!

[y1015860449]

Crash core:

#0  0x00007fa48b4d3277 in raise () from /lib64/libc.so.6
#1  0x00007fa48b4d4968 in abort () from /lib64/libc.so.6
#2  0x00007fa48b515d37 in __libc_message () from /lib64/libc.so.6
#3  0x00007fa48b51c5e4 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fa48b521c39 in _int_realloc () from /lib64/libc.so.6
#5  0x00007fa48b522eb2 in realloc () from /lib64/libc.so.6
#6  0x00007fa48d2b5649 in CRYPTO_realloc () from ../lib/libcrypto.so.10
#7  0x00007fa48d330b11 in lh_insert () from ../lib/libcrypto.so.10
#8  0x00007fa48d3331e4 in int_thread_set_item () from ../lib/libcrypto.so.10
#9  0x00007fa48d333d6e in ERR_get_state () from ../lib/libcrypto.so.10
#10 0x00007fa48d333ef5 in ERR_clear_error () from ../lib/libcrypto.so.10
#11 0x00007fa48d6665b5 in ssl23_connect () from ../lib/libssl.so.10
#12 0x000000000049add6 in httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}::operator()(ssl_st*) const ()
#13 0x000000000049ed1b in bool httplib::detail::read_and_close_socket_ssl<httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#2}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*, httplib::Stream&, bool, bool&)#3}>(int, unsigned long, ssl_ctx_st*, std::mutex&, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#1}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*)#2}, httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&)::{lambda(ssl_st*, httplib::Stream&, bool, bool&)#3}) ()
#14 0x000000000049afb0 in httplib::SSLClient::read_and_close_socket(int, httplib::Request&, httplib::Response&) ()
#15 0x0000000000499c88 in httplib::Client::send(httplib::Request&, httplib::Response&) ()
#16 0x000000000049a57e in httplib::Client::Post(char const*, std::multimap<std::string, std::string, httplib::detail::ci, std::allocator<std::pair<std::string const, std::string> > > const&, std::string const&, char const*) ()
#17 0x00000000004d8aad in CMiPushClient::pushDataHandle(im::ANDPushMsg, bool) ()
#18 0x00000000004da364 in void std::_Mem_fn<void (CMiPushClient::*)(im::ANDPushMsg, bool)>::operator()<im::ANDPushMsg&, bool&, void>(CMiPushClient*, im::ANDPushMsg&, bool&) const ()
#19 0x00000000004da26a in void std::_Bind<std::_Mem_fn<void (CMiPushClient::*)(im::ANDPushMsg, bool)> (CMiPushClient*, im::ANDPushMsg, bool)>::__call<void, , 0ul, 1ul, 2ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul>) ()
#20 0x00000000004d9f74 in void std::_Bind<std::_Mem_fn<void (CMiPushClient::*)(im::ANDPushMsg, bool)> (CMiPushClient*, im::ANDPushMsg, bool)>::operator()<, void>() ()
---Type <return> to continue, or q <return> to quit---
#21 0x00000000004d9de0 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (CMiPushClient::*)(im::ANDPushMsg, bool)> (CMiPushClient*, im::ANDPushMsg, bool)> >::_M_invoke(std::_Any_data const&) ()
#22 0x000000000049c250 in std::function<void ()>::operator()() const ()
#23 0x0000000000493bd1 in CThreadPool::run_task() ()
#24 0x00000000004d3ed1 in void std::_Mem_fn<void (CThreadPool::*)()>::operator()<, void>(CThreadPool*) const ()
#25 0x00000000004d3e3a in void std::_Bind<std::_Mem_fn<void (CThreadPool::*)()> (CThreadPool*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) ()
#26 0x00000000004d3d00 in void std::_Bind<std::_Mem_fn<void (CThreadPool::*)()> (CThreadPool*)>::operator()<, void>() ()
#27 0x00000000004d3b88 in void std::_Bind_simple<std::_Bind<std::_Mem_fn<void (CThreadPool::*)()> (CThreadPool*)> ()>::_M_invoke<>(std::_Index_tuple<>) ()
#28 0x00000000004d39a1 in std::_Bind_simple<std::_Bind<std::_Mem_fn<void (CThreadPool::*)()> (CThreadPool*)> ()>::operator()() ()
#29 0x00000000004d352e in std::thread::_Impl<std::_Bind_simple<std::_Bind<std::_Mem_fn<void (CThreadPool::*)()> (CThreadPool*)> ()> >::_M_run() ()
#30 0x00007fa48be37c70 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../gcc-4.9.2/libstdc++-v3/src/c++11/thread.cc:84
#31 0x00007fa48dab3e25 in start_thread () from /lib64/libpthread.so.0
#32 0x00007fa48b59bbad in clone () from /lib64/libc.so.6

code:

void CMiPushClient::addMiPushData(std::shared_ptr<im::ANDPushMsg> msg)
{
	m_threadPool.add_task(&CMiPushClient::pushDataHandle, this, msg, false);
}

void CMiPushClient::pushDataHandle(std::shared_ptr<im::ANDPushMsg> msg, bool bGlobal)
{
	std::string sPostData = getPostData(msg);
	std::string sUrl = bGlobal ? mi_global_url : mi_url;
	httplib::SSLClient client(mi_url.c_str(), 443, 5);
	auto res = client.Post(mi_fun.c_str(), m_header, sPostData.c_str(), "application/x-www-form-urlencoded");
...............
}

[yhirose]

@y1015860449, thanks for the code snippet. It helps me a lot. I have some additional simple questions for you, so that I could reproduce it on my machine:

1. It rarely happens, right? (four crashes out of thousands of requests in two days, right?)
2. OpenSSL library version (1.0.2, 1.1.0, or 1.1.1?)
3. OS (Linux, MacOS or Windows?)

I especially want to know the answer to the question 2. Thanks.

[yhirose]

https://www.openssl.org/blog/blog/2017/02/21/threads/
https://stackoverflow.com/questions/3919420/tutorial-on-using-openssl-with-pthreads

[y1015860449]

@yhirose 1、yes。
2、OpenSSL 1.0.1e
3、centos7

[yhirose]

@y1015860449, thanks for the answers. I now have to work seriously on the thread issue which happens with versions prior to 1.1.0.

cpp-httplib/httplib.h

Lines 2361 to 2365 in 3d1ae3a

	read_and_close_socket_ssl(socket_t sock, size_t keep_alive_max_count,
	// TODO: OpenSSL 1.0.2 occasionally crashes...
	// The upcoming 1.1.0 is going to be thread safe.
	SSL_CTX *ctx, std::mutex &ctx_mutex,
	U SSL_connect_or_accept, V setup, T callback) {

[yhirose]

@y1015860449, I made a necessary thread safety change for versions prior to 1.1.0 according to the OpenSSL documentation. Could you try with the latest httplib.h?

[y1015860449]

@yhirose I will test it today. Does OPENSSL_VERSION_NUMBER require conditional compilation? How to compile with CMakeList if needed?

[yhirose]

@y1015860449, as you can see my changes below, I have already added the #if OPENSSL_VERSION_NUMBER < 0x10100000L condition. So all you need to do is just to replace with the latest httplib.h. Thanks for taking time!

cpp-httplib/httplib.h

Lines 2484 to 2523 in 7267b3f

	#if OPENSSL_VERSION_NUMBER < 0x10100000L
	class SSLThreadLocks {
	public:
	SSLThreadLocks() {
	CRYPTO_set_locking_callback(locking_callback);
	}
	~SSLThreadLocks() {
	CRYPTO_set_locking_callback(nullptr);
	}
	private:
	static void locking_callback(int mode, int type, const char * /*file*/, int /*line*/) {
	if (mode & CRYPTO_LOCK) {
	locks_[type].lock();
	} else {
	locks_[type].unlock();
	}
	}
	static std::vector<std::mutex> locks_;
	};
	std::vector<std::mutex> SSLThreadLocks::locks_(CRYPTO_num_locks());
	#endif
	class SSLInit {
	public:
	SSLInit() {
	SSL_load_error_strings();
	SSL_library_init();
	}
	~SSLInit() { ERR_free_strings(); }
	private:
	#if OPENSSL_VERSION_NUMBER < 0x10100000L
	SSLThreadLocks thread_init_;
	#endif
	};

[y1015860449]

@yhirose Compile Error.

CMakeFiles/apush_server.dir/hw_push_client.cpp.o:(.bss+0x0): multiple definition of `httplib::detail::SSLThreadLocks::locks_'
CMakeFiles/apush_server.dir/google_push_client.cpp.o:(.bss+0x0): first defined here
CMakeFiles/apush_server.dir/imappframe.cpp.o:(.bss+0x0): multiple definition of `httplib::detail::SSLThreadLocks::locks_'
CMakeFiles/apush_server.dir/google_push_client.cpp.o:(.bss+0x0): first defined here
CMakeFiles/apush_server.dir/imframe.cpp.o:(.bss+0x0): multiple definition of `httplib::detail::SSLThreadLocks::locks_'
CMakeFiles/apush_server.dir/google_push_client.cpp.o:(.bss+0x0): first defined here
CMakeFiles/apush_server.dir/mi_push_client.cpp.o:(.bss+0x0): multiple definition of `httplib::detail::SSLThreadLocks::locks_'
CMakeFiles/apush_server.dir/google_push_client.cpp.o:(.bss+0x0): first defined here
CMakeFiles/apush_server.dir/push_provide_server.cpp.o:(.bss+0x0): multiple definition of `httplib::detail::SSLThreadLocks::locks_'
CMakeFiles/apush_server.dir/google_push_client.cpp.o:(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status
make[2]: *** [apush_server] Error 1
make[1]: *** [CMakeFiles/apush_server.dir/all] Error 2
make: *** [all] Error 2

[PixlRainbow]

Were you using #include inside files which were then themselves included? Or is cmake cache still using the old version of cpp-httplib?