Application(redis client) crashes with SIGSEGV at redisBufferRead at hiredis.c:941 while making a GET call to redis server

[Sukesh0513]

PROBLEM

In our application we use c++, so we use used redis++(1.3.7) and hired(1.0.2) for our redis client implementation, In the application we make a GET call to redis server, then after that call, a bunch of functions in redis++ and hiredis are called one by one, and the application finally crashed with SIGSEGV while calling the function redisBufferRead at at hiredis.c:941. This is what I could see from backtracing the corefile.

Note: in our application we do several GET call until we get the desired value the from redis-server, in some cases if the value is configured in redis server, we will have a null value as return and we will retry the GET call again until we get a non null value. In the initial stage, redis server is not populated this value and our application will keep on making get call but it seals like this issue is not happening for every get call, it only happen once after multiple retries.

here is the backTrace from the corefile

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f6dc0b49a21 in redisBufferRead (c=0x1720680) at hiredis.c:941
941 hiredis.c: No such file or directory.

#0 0x00007f6dc0b49a21 in redisBufferRead (c=0x1720680) at hiredis.c:941
redis/hiredis#1 0x00007f6dc0b49d78 in redisGetReply (c=c@entry=0x1720680, reply=reply@entry=0x7f6da0989618) at hiredis.c:1032
redis/hiredis#2 0x00007f6dc0b84b66 in redis::Connection::recv (this=this@entry=0x7f6da0989688, handle_error_reply=handle_error_reply@entry=true)
at /tmp/tmp.PiajinzKq4/redis-plus-plus-1.3.7/src/redis++/connection.cpp:212
redis/hiredis#3 0x00007f6dc0b9e913 in redis::Redis::_command<void ()(redis::Connection&, std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > const&> (this=, cmd=0x7f6dc0b9cb40 <redis::cmd::get(redis::Connection&, std::basic_string_view<char, std::char_traits > const&)>, connection=...)
at /tmp/tmp.PiajinzKq4/redis-plus-plus-1.3.7/src/redis++/redis.hpp:1316
redis/hiredis#4 redis::Redis::command<void ()(redis::Connection&, std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > const&> (
this=, cmd=cmd@entry=0x7f6dc0b9cb40 <redis::cmd::get(redis::Connection&, std::basic_string_view<char, std::char_traits > const&)>)
at /tmp/tmp.PiajinzKq4/redis-plus-plus-1.3.7/src/redis++/redis.hpp:47
redis/hiredis#5 0x00007f6dc0b9733c in redis::Redis::get[abi:cxx11](std::basic_string_view<char, std::char_traits > const&) (this=, key=...)
at /tmp/tmp.PiajinzKq4/redis-plus-plus-1.3.7/src/redis++/redis.cpp:314

Initially I thought that the redisContext which is used in the function redisBufferRead at hiredis.c:941 is pointing to a null pointer or might be not be initialized but when the checked the backtrack and tried to print it, I came to know that it is not null and is initialized, looking the back trace I do not understand or can find any reson for redisBufferRead to throw a SIGSEGV. Any idea or understand on why this could happen will be very helpful.

The only reason that I can think of is, ctx which is initialized at line 236 in github.com__sewenew__redis-plus-plus/src/sw/redis++/connection.cpp is being dereferenced by the time it was accessed at line 241. does this happen earlier or are there any other reason for this to happen.

ENVIRONMENT
OS: suse linux
version: 15-SP5
hiredis: 1.0.2
redis: 1.3.7

[sewenew]

Both hiredis 1.0.2 and redis-plus-plus 1.3.7 are very old version, and the latest version might have fixed bugs that caused your problem. I'd suggest you upgrade them to the latest version to have a try. Also please give me a minimum code snippet that can reproduce your problem.

Also please ensure there's only ONE version of hiredis is installed. Otherwise, you might get some wired problems.

Regards

[Sukesh0513]

Hi again,

Thanks for the suggestion, we have uplifted the redis++(1.3.11) and hiredis(1.2.0) but even with the new versions we still see this problem.

[sewenew]

I cannot reproduce your problem by sending GET command to Redis in a for loop. Can you give me a minimum code snippet that can reproduce your problem, so that I can do some debugging.

[Sukesh0513]

Hi,

Thanks for looking into it, we came across this problem while running it in k8s cluster(redis server is one pod and our application is another pod which is a client to redis pod), these are private docker images and I am afraid that I can not share you the image. But we just do a GET command in a loop to wards redis server as you just tested.

I will try to build a small k8s setup from opensource docker images and see if can reproduce the issue, may be then you can debug it.

[albert3021]

I don't know why. But it seems the same problem when first invoke method into instance of class Redis under the Ubuntu 22.04 LTS x86_64 environment.
The related library version are: libhiredis.so.1.1.0 and libredis++.so.1.3.13. All build from recent source tag branch with GCC 11.4.0.

The strange things is when I build the same code under win11 with MSVS or cross-compile to Armv7-linux platform, it goes well.

Below is the backtrace from gdb, hope it will provide some help.

#Thread 12 "dfmain" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6748640 (LWP 2998)]
0x00007ffff6fed0bd in redisBufferRead () from /usr/local/lib/libhiredis.so.1.1.0
(gdb) bt
#0 0x00007ffff6fed0bd in redisBufferRead () from /usr/local/lib/libhiredis.so.1.1.0
#1 0x00007ffff6fed4c4 in redisGetReply () from /usr/local/lib/libhiredis.so.1.1.0
#2 0x00007ffff7036068 in sw::redis::Connection::recv(bool) () from /usr/local/lib/libredis++.so.1
#3 0x00007ffff703635f in sw::redis::Connection::_auth() () from /usr/local/lib/libredis++.so.1
#4 0x00007ffff703666d in sw::redis::Connection::_set_options() () from /usr/local/lib/libredis++.so.1
#5 0x00007ffff703693c in sw::redis::Connection::Connection(sw::redis::ConnectionOptions const&) ()
from /usr/local/lib/libredis++.so.1
#6 0x00007ffff7036c03 in sw::redis::Connection::reconnect() () from /usr/local/lib/libredis++.so.1
#7 0x00007ffff7039604 in sw::redis::ConnectionPool::fetch() () from /usr/local/lib/libredis++.so.1
#8 0x00007ffff704a62b in std::enable_if<!std::is_convertible<void ()(sw::redis::Connection&, std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > >::value, std::unique_ptr<redisReply, sw::redis::ReplyDeleter> >::type sw::redis::Redis::command<void ()(sw::redis::Connection&, std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > const&>(void (*)(sw::redis::Connection&, std::basic_string_view<char, std::char_traits > const&), std::basic_string_view<char, std::char_traits > const&) () from /usr/local/lib/libredis++.so.1
#9 0x00007ffff703de8f in sw::redis::Redis::echo[abi:cxx11](std::basic_string_view<char, std::char_traits > const&) () from /usr/local/lib/libredis++.so.1
#10 0x00007ffff71c8f95 in RedisAgentContext::Initialize (this=0x7fffdc0019a0) at src/dfl2redissta.cpp:50
#11 0x00007ffff7182d61 in CRedisAgentDriver::CRedisAgentThread::Run (this=0x555555700460) at src/dfl2redisdrv.cpp:737
#12 0x00007ffff7c808f5 in GX_THREAD::ThreadEntryRoutine (this=0x555555667cc0, pThd=0x555555701360)
at extra/glbthread.cpp:32
#13 0x00007ffff7c804b1 in IThreadObject::apiThreadStartRoutine (arg=0x555555701360) at extra/glbthrapiw.cpp:620
#14 0x00007ffff7849ac3 in start_thread (arg=) at ./nptl/pthread_create.c:442
#15 0x00007ffff78db850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)

[albert3021]

Finally I got this problem down. The problem is mainly due to stack overflow in libhiredis library. Also in redis++ library may have slight mistakes. With different compiler & programing invoke structure, it shows werid result. Below shows the fix:

1. find "Connection::recv" method in "connection.cpp" of the redis++ project, modify "auto *ctx = _context()" might be "auot ctx = _context()";
2. find "redisBufferRead" function in "hiredis.c" of hiredis project, modify "char buf[1024*16]" with "char buf[1024]" or even small buffer size;
   Generally to say, it might not be best fix. Define too BIG buffer size inside function stack of C library is not a good idea. Maybe should consider memory pool or as a paramter. Hope it'll help.