[CRASH] Keydb crashes with redisearch in KeyDB image created from main branch

[sapvs]

Keydb crashes with redisearch in KeyDB image created from main branch

Crash report

1:1:M 10 Aug 2023 11:16:53.790 # Server initialized
1:1:M 10 Aug 2023 11:16:53.794 * <search> Redis version found by RedisSearch : 255.255.255 - oss
1:1:M 10 Aug 2023 11:16:53.794 * <search> RediSearch version 2.8.1 (Git=HEAD-251cc55f)
1:1:M 10 Aug 2023 11:16:53.794 * <search> Low level api version 1 initialized successfully
1:1:M 10 Aug 2023 11:16:53.794 * <search> concurrent writes: OFF, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (
ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search re
sults:  1000000, search pool size: 20, index pool size: 8,
1:1:M 10 Aug 2023 11:16:53.797 * <search> Initialized thread pools!
1:1:M 10 Aug 2023 11:16:53.797 * <search> Enabled diskless replication
1:1:M 10 Aug 2023 11:16:53.797 * <search> Enabled role change notification
1:1:M 10 Aug 2023 11:16:53.797 * Module 'search' loaded from /etc/libs/redisearch.so


=== KEYDB BUG REPORT START: Cut & paste starting from here ===
1:1:M 10 Aug 2023 11:16:53.797 # KeyDB 255.255.255 crashed by signal: 11, si_code: 1
1:1:M 10 Aug 2023 11:16:53.797 # Accessing address: 0x48
1:1:M 10 Aug 2023 11:16:53.797 # Crashed running the instruction at: 0x7efc64c908e6

------ STACK TRACE ------
EIP:
/etc/libs/redisearch.so(+0x1a78e6) [0x7efc64c908e6]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7efc67923420]
/etc/libs/redisearch.so(+0x1a78e6) [0x7efc64c908e6]
keydb-server *:6379(+0x26aef5) [0x55a6880eeef5]
keydb-server *:6379(InitServerLast()+0x6e) [0x55a6881c28de]
keydb-server *:6379(main+0x566) [0x55a68808f096]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7efc67741083]
keydb-server *:6379(_start+0x2e) [0x55a6880b467e]

------ REGISTERS ------
1:1:M 10 Aug 2023 11:16:53.798 #
RAX:0000000000000000 RBX:0000000000000003
RCX:0000000000000003 RDX:0000000000000001
RDI:00007ffff555a1c0 RSI:0000000000000003
RBP:00007ffff555a1c0 RSP:00007ffff555a120
R8 :0000000000000000 R9 :0000000000001000
R10:000055a688730b5d R11:0000000000000000
R12:00007efc64fff098 R13:000055a68888e8f8
R14:00007ffff555a1c0 R15:00007efc66495960
RIP:00007efc64c908e6 EFL:0000000000010246
CSGSFS:002b000000000033
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a12f) -> 0000000000000000
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a12e) -> 0000000000000000
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a12d) -> 0000000000000000
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a12c) -> 00007efc65dc0c00
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a12b) -> 00007efc664684e0
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a12a) -> 000055a688102be0
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a129) -> 000055a6880f271d
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a128) -> 00007efc65d93900
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a127) -> 00007efc00000000
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a126) -> 0000000000000000
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a125) -> 010055a600000003
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a124) -> 0000000066480781
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a123) -> 000055a6880eeef5
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a122) -> 0000000000000000
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a121) -> 00007ffff555a150
1:1:M 10 Aug 2023 11:16:53.798 # (00007ffff555a120) -> 0000000000000003

------ INFO OUTPUT ------
# Server
redis_version:255.255.255
redis_git_sha1:441b74a9
redis_git_dirty:1
redis_build_id:e4a8c2954cbe3756
redis_mode:standalone
os:Linux 5.10.102.1-microsoft-standard-WSL2 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:9.4.0
process_id:1
process_supervised:no
run_id:5037c93d4dbb62b46a23b9139ccf4685e74e7718
tcp_port:6379
server_time_usec:1691666213042171
uptime_in_seconds:0
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:13944613
executable:/data/keydb-server
config_file:
availability_zone:

# Clients
connected_clients:0
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
current_client_thread:0
thread_0_clients:0

# Memory
used_memory:1662584
used_memory_human:1.59M
used_memory_rss:0
used_memory_rss_human:0B
used_memory_peak:1662584
used_memory_peak_human:1.59M
used_memory_peak_perc:101.74%
used_memory_overhead:0
used_memory_startup:0
used_memory_dataset:1662584
used_memory_dataset_perc:100.00%
allocator_allocated:0
allocator_active:0
allocator_resident:0
total_system_memory:4114530304
total_system_memory_human:3.83G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:-nan
allocator_frag_bytes:0
allocator_rss_ratio:-nan
allocator_rss_bytes:0
rss_overhead_ratio:-nan
rss_overhead_bytes:0
mem_fragmentation_ratio:-nan
mem_fragmentation_bytes:0
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
storage_provider:none
available_system_memory:unavailable

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1691666213
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:0
total_commands_processed:1
instantaneous_ops_per_sec:0
total_net_input_bytes:0
total_net_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:0
total_writes_processed:0
instantaneous_lock_contention:1
avg_lock_contention:0.000000
storage_provider_read_hits:0
storage_provider_read_misses:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:02d80df7bcf3ccb3426f03930fb7e993631fd29b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.021010
used_cpu_user:0.021010
used_cpu_sys_children:0.000000
used_cpu_user_children:0.002751
server_threads:1
long_lock_waits:0
used_cpu_sys_main_thread:0.019326
used_cpu_user_main_thread:0.019326

# Modules
module:name=search,ver=20801,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors]

# Commandstats
cmdstat_info:calls=1,usec=28,usec_per_call=28.00,rejected_calls=0,failed_calls=0

# Errorstats

# Cluster
cluster_enabled:0

# Keyspace

# KeyDB
mvcc_depth:0

------ CLIENT LIST OUTPUT ------

------ MODULES INFO OUTPUT ------
# search_version
search_version:2.8.1
search_redis_version:255.255.255 - oss

# search_index
search_number_of_indexes:0

# search_fields_statistics

# search_dialect_statistics
search_dialect_1:0
search_dialect_2:0
search_dialect_3:0

# search_runtime_configurations
search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:1000000
search_max_aggregate_results:-1
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3

------ FAST MEMORY TEST ------
*** Preparing to test memory region 55a688876000 (2334720 bytes)
*** Preparing to test memory region 7efc642e9000 (8388608 bytes)
*** Preparing to test memory region 7efc64ff9000 (28672 bytes)
*** Preparing to test memory region 7efc65000000 (4194304 bytes)
*** Preparing to test memory region 7efc65580000 (8388608 bytes)
*** Preparing to test memory region 7efc65d80000 (11010048 bytes)
*** Preparing to test memory region 7efc668d8000 (69632 bytes)
*** Preparing to test memory region 7efc6691c000 (32768 bytes)
*** Preparing to test memory region 7efc66a9a000 (4096 bytes)
*** Preparing to test memory region 7efc66ad7000 (8192 bytes)
*** Preparing to test memory region 7efc66b35000 (4096 bytes)
*** Preparing to test memory region 7efc66c6e000 (4096 bytes)
*** Preparing to test memory region 7efc66c7b000 (8192 bytes)
*** Preparing to test memory region 7efc66e55000 (8192 bytes)
*** Preparing to test memory region 7efc66e71000 (8192 bytes)
*** Preparing to test memory region 7efc66eb9000 (4096 bytes)
*** Preparing to test memory region 7efc66f97000 (8192 bytes)
*** Preparing to test memory region 7efc67262000 (8192 bytes)
*** Preparing to test memory region 7efc673f4000 (8192 bytes)
*** Preparing to test memory region 7efc6745b000 (8192 bytes)
*** Preparing to test memory region 7efc6754b000 (8192 bytes)
*** Preparing to test memory region 7efc6763e000 (8192 bytes)
*** Preparing to test memory region 7efc6790b000 (16384 bytes)
*** Preparing to test memory region 7efc6792e000 (16384 bytes)
*** Preparing to test memory region 7efc6794d000 (8192 bytes)
*** Preparing to test memory region 7efc67c7d000 (12288 bytes)
*** Preparing to test memory region 7efc67f52000 (16384 bytes)
*** Preparing to test memory region 7efc6800b000 (8192 bytes)
*** Preparing to test memory region 7efc6803f000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /etc/libs/redisearch.so (base 0x7efc64ae9000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------

Aditional information

1. OS distribution and version

Ubuntu 20.04 docker image
KeyDB image created from main branch.

1. Steps to reproduce (if any)
   Created KeyDB image using pkg/docker/Dockerfile
   docker build --build-arg BRANCH=main -t keydb_local .

Compiled RediSearch, copied binary to keydb image in above and started the container.

Dockerfile

FROM ubuntu:20.04 AS builder 

WORKDIR /keydb
RUN apt update
RUN apt install -y git make python3 build-essential make
RUN git clone -b v2.8.1 --recursive https://github.com/RediSearch/RediSearch.git
RUN cd RediSearch && make setup  && make build 

FROM keydb_local:latest AS prod 
COPY --from=builder /keydb/RediSearch/bin/linux-x64-release/search/redisearch.so /etc/libs/redisearch.so
CMD [ "--loadmodule", "/etc/libs/redisearch.so" ]

Running this image causes failure. However when CMD loadmodule is removed from the Dockerfile. And loading module from keydb-cli causes no crash.

Module is loaded and commands are available

127.0.0.1:6379> ft._list
(error) ERR unknown command `ft._list`, with args beginning with:
127.0.0.1:6379> module load /etc/libs/redisearch.so
OK
127.0.0.1:6379> ft._list
(empty array)
127.0.0.1:6379> ft._list
1) "idx:bonus"

[CanRau]

Just tried, though using this Dockerfile

FROM ubuntu:20.04 AS builder 

WORKDIR /keydb
RUN apt-get update
RUN apt-get install -y git make python3 build-essential make
RUN git clone -b v2.8.1 --recursive https://github.com/RediSearch/RediSearch.git
RUN cd RediSearch && make setup  && make build 

FROM eqalpha/keydb:latest AS prod 
ADD keydb.conf /etc/keydb.conf
COPY --from=builder /keydb/RediSearch/bin/linux-x64-release/search/redisearch.so /etc/libs/redisearch.so
CMD ["keydb-server", "/etc/keydb.conf"]

and following keydb.conf

dir /data/ 
protected-mode no 
appendonly yes

save 900 1
save 300 10
save 60 10000

active-replica yes
replica-read-only no
multi-master yes
replica-serve-stale-data yes

maxmemory-policy noeviction

loadmodule /etc/libs/redisearch.so

most of the config comes from my currently running cluster, I just remembered that I read somewhere that CMD [ "--loadmodule", "/etc/libs/redisearch.so" ] might fail

Though I'm getting the following error, I think I got this in former trials as well

# Configuration loaded
* monotonic clock: POSIX clock_gettime
* Running mode=standalone, port=6379.
# Server initialized
# Module /etc/libs/redisearch.so failed to load: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /etc/libs/redisearch.so)

also commented in the forum, though seems to be down or having certificate issues currently or sthg

Really would love to get RediSearch and RedisJSON working, that'd be 🔥

I think it would be really nice to be able to get the Redis modules from the redis-stack-server Dockerfile, instead of manually building them from source, though not sure that's going to work, I tried that once but failed as well, can share my Dockerfile later.

Just realized that I also commented here #406

I understand that KeyDB is open source, which I highly appreciate and value, though it's a little worrying how little official reactions issues here get, same feels on Slack and the forums 🥲

[CanRau]

Just now saw your comments on Slack and that you're using that Dockerfile because of the newer Ubuntu so tried as well, all went well until I started a container which resulted in basically the same as OP I think

Bug report

2023-08-16 13:11:45 === KEYDB BUG REPORT START: Cut & paste starting from here ===
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.851 # KeyDB 255.255.255 crashed by signal: 11, si_code: 1
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.851 # Accessing address: 0x48
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.851 # Crashed running the instruction at: 0x7fae680108e6
2023-08-16 13:11:45 
2023-08-16 13:11:45 ------ STACK TRACE ------
2023-08-16 13:11:45 EIP:
2023-08-16 13:11:45 /etc/libs/redisearch.so(+0x1a78e6) [0x7fae680108e6]
2023-08-16 13:11:45 
2023-08-16 13:11:45 Backtrace:
2023-08-16 13:11:45 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fae6a041420]
2023-08-16 13:11:45 /etc/libs/redisearch.so(+0x1a78e6) [0x7fae680108e6]
2023-08-16 13:11:45 keydb-server *:6379(+0x26aef5) [0x55d7f5ef1ef5]
2023-08-16 13:11:45 keydb-server *:6379(InitServerLast()+0x6e) [0x55d7f5fc58de]
2023-08-16 13:11:45 keydb-server *:6379(main+0x566) [0x55d7f5e92096]
2023-08-16 13:11:45 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fae69e5f083]
2023-08-16 13:11:45 keydb-server *:6379(_start+0x2e) [0x55d7f5eb767e]
2023-08-16 13:11:45 
2023-08-16 13:11:45 ------ REGISTERS ------
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # 
2023-08-16 13:11:45 RAX:0000000000000000 RBX:0000000000000003
2023-08-16 13:11:45 RCX:0000000000000003 RDX:0000000000000001
2023-08-16 13:11:45 RDI:00007ffdd82f92c0 RSI:0000000000000003
2023-08-16 13:11:45 RBP:00007ffdd82f92c0 RSP:00007ffdd82f9220
2023-08-16 13:11:45 R8 :0000000000000000 R9 :0000000000001000
2023-08-16 13:11:45 R10:000055d7f6533b5d R11:0000000000000246
2023-08-16 13:11:45 R12:00007fae6837f098 R13:000055d7f66918f8
2023-08-16 13:11:45 R14:00007ffdd82f92c0 R15:00007fae68a94820
2023-08-16 13:11:45 RIP:00007fae680108e6 EFL:0000000000010246
2023-08-16 13:11:45 CSGSFS:002b000000000033
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f922f) -> 0000000000000000
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f922e) -> 0000000000000000
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f922d) -> 0000000000000000
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f922c) -> 00007fae683d29c0
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f922b) -> 00007fae68a684e0
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f922a) -> 000055d7f5f05be0
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f9229) -> 000055d7f5ef571d
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f9228) -> 00007fae683c3400
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f9227) -> 00007fae00000000
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f9226) -> 0000000000000000
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f9225) -> 010055d700000003
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f9224) -> 0000000068a807a1
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f9223) -> 000055d7f5ef1ef5
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f9222) -> 0000000000000000
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f9221) -> 00007ffdd82f9250
2023-08-16 13:11:45 1:1:M 16 Aug 2023 18:11:45.852 # (00007ffdd82f9220) -> 0000000000000003
2023-08-16 13:11:45 
2023-08-16 13:11:45 ------ INFO OUTPUT ------
2023-08-16 13:11:45 # Server
2023-08-16 13:11:45 redis_version:255.255.255
2023-08-16 13:11:45 redis_git_sha1:441b74a9
2023-08-16 13:11:45 redis_git_dirty:1
2023-08-16 13:11:45 redis_build_id:82759f545caa9a17
2023-08-16 13:11:45 redis_mode:standalone
2023-08-16 13:11:45 os:Linux 5.15.49-linuxkit-pr x86_64
2023-08-16 13:11:45 arch_bits:64
2023-08-16 13:11:45 multiplexing_api:epoll
2023-08-16 13:11:45 atomicvar_api:atomic-builtin
2023-08-16 13:11:45 gcc_version:9.4.0
2023-08-16 13:11:45 process_id:1
2023-08-16 13:11:45 process_supervised:no
2023-08-16 13:11:45 run_id:4761a7974aa8aa6cb46ed411ffdd5f89b3f57a9b
2023-08-16 13:11:45 tcp_port:6379
2023-08-16 13:11:45 server_time_usec:1692209505784547
2023-08-16 13:11:45 uptime_in_seconds:0
2023-08-16 13:11:45 uptime_in_days:0
2023-08-16 13:11:45 hz:10
2023-08-16 13:11:45 configured_hz:10
2023-08-16 13:11:45 lru_clock:14487905
2023-08-16 13:11:45 executable:/data/keydb-server
2023-08-16 13:11:45 config_file:/etc/keydb.conf
2023-08-16 13:11:45 availability_zone:
2023-08-16 13:11:45 
2023-08-16 13:11:45 # Clients
2023-08-16 13:11:45 connected_clients:0
2023-08-16 13:11:45 cluster_connections:0
2023-08-16 13:11:45 maxclients:10000
2023-08-16 13:11:45 client_recent_max_input_buffer:0
2023-08-16 13:11:45 client_recent_max_output_buffer:0
2023-08-16 13:11:45 blocked_clients:0
2023-08-16 13:11:45 tracking_clients:0
2023-08-16 13:11:45 clients_in_timeout_table:0
2023-08-16 13:11:45 current_client_thread:0
2023-08-16 13:11:45 thread_0_clients:0
2023-08-16 13:11:45 
2023-08-16 13:11:45 # Memory
2023-08-16 13:11:45 used_memory:1743832
2023-08-16 13:11:45 used_memory_human:1.66M
2023-08-16 13:11:45 used_memory_rss:0
2023-08-16 13:11:45 used_memory_rss_human:0B
2023-08-16 13:11:45 used_memory_peak:1743832
2023-08-16 13:11:45 used_memory_peak_human:1.66M
2023-08-16 13:11:45 used_memory_peak_perc:100.28%
2023-08-16 13:11:45 used_memory_overhead:8
2023-08-16 13:11:45 used_memory_startup:0
2023-08-16 13:11:45 used_memory_dataset:1743824
2023-08-16 13:11:45 used_memory_dataset_perc:100.00%
2023-08-16 13:11:45 allocator_allocated:0
2023-08-16 13:11:45 allocator_active:0
2023-08-16 13:11:45 allocator_resident:0
2023-08-16 13:11:45 total_system_memory:8346099712
2023-08-16 13:11:45 total_system_memory_human:7.77G
2023-08-16 13:11:45 used_memory_lua:37888
2023-08-16 13:11:45 used_memory_lua_human:37.00K
2023-08-16 13:11:45 used_memory_scripts:0
2023-08-16 13:11:45 used_memory_scripts_human:0B
2023-08-16 13:11:45 number_of_cached_scripts:0
2023-08-16 13:11:45 maxmemory:0
2023-08-16 13:11:45 maxmemory_human:0B
2023-08-16 13:11:45 maxmemory_policy:noeviction
2023-08-16 13:11:45 allocator_frag_ratio:-nan
2023-08-16 13:11:45 allocator_frag_bytes:0
2023-08-16 13:11:45 allocator_rss_ratio:-nan
2023-08-16 13:11:45 allocator_rss_bytes:0
2023-08-16 13:11:45 rss_overhead_ratio:-nan
2023-08-16 13:11:45 rss_overhead_bytes:0
2023-08-16 13:11:45 mem_fragmentation_ratio:-nan
2023-08-16 13:11:45 mem_fragmentation_bytes:0
2023-08-16 13:11:45 mem_not_counted_for_evict:4
2023-08-16 13:11:45 mem_replication_backlog:0
2023-08-16 13:11:45 mem_clients_slaves:0
2023-08-16 13:11:45 mem_clients_normal:0
2023-08-16 13:11:45 mem_aof_buffer:8
2023-08-16 13:11:45 mem_allocator:jemalloc-5.2.1
2023-08-16 13:11:45 active_defrag_running:0
2023-08-16 13:11:45 lazyfree_pending_objects:0
2023-08-16 13:11:45 lazyfreed_objects:0
2023-08-16 13:11:45 storage_provider:none
2023-08-16 13:11:45 available_system_memory:unavailable
2023-08-16 13:11:45 
2023-08-16 13:11:45 # Persistence
2023-08-16 13:11:45 loading:0
2023-08-16 13:11:45 current_cow_size:0
2023-08-16 13:11:45 current_cow_size_age:0
2023-08-16 13:11:45 current_fork_perc:0.00
2023-08-16 13:11:45 current_save_keys_processed:0
2023-08-16 13:11:45 current_save_keys_total:0
2023-08-16 13:11:45 rdb_changes_since_last_save:0
2023-08-16 13:11:45 rdb_bgsave_in_progress:0
2023-08-16 13:11:45 rdb_last_save_time:1692209505
2023-08-16 13:11:45 rdb_last_bgsave_status:ok
2023-08-16 13:11:45 rdb_last_bgsave_time_sec:-1
2023-08-16 13:11:45 rdb_current_bgsave_time_sec:-1
2023-08-16 13:11:45 rdb_last_cow_size:0
2023-08-16 13:11:45 aof_enabled:1
2023-08-16 13:11:45 aof_rewrite_in_progress:0
2023-08-16 13:11:45 aof_rewrite_scheduled:0
2023-08-16 13:11:45 aof_last_rewrite_time_sec:-1
2023-08-16 13:11:45 aof_current_rewrite_time_sec:-1
2023-08-16 13:11:45 aof_last_bgrewrite_status:ok
2023-08-16 13:11:45 aof_last_write_status:ok
2023-08-16 13:11:45 aof_last_cow_size:0
2023-08-16 13:11:45 module_fork_in_progress:0
2023-08-16 13:11:45 module_fork_last_cow_size:0
2023-08-16 13:11:45 aof_current_size:0
2023-08-16 13:11:45 aof_base_size:0
2023-08-16 13:11:45 aof_pending_rewrite:0
2023-08-16 13:11:45 aof_buffer_length:0
2023-08-16 13:11:45 aof_rewrite_buffer_length:0
2023-08-16 13:11:45 aof_pending_bio_fsync:0
2023-08-16 13:11:45 aof_delayed_fsync:0
2023-08-16 13:11:45 
2023-08-16 13:11:45 # Stats
2023-08-16 13:11:45 total_connections_received:0
2023-08-16 13:11:45 total_commands_processed:2
2023-08-16 13:11:45 instantaneous_ops_per_sec:0
2023-08-16 13:11:45 total_net_input_bytes:0
2023-08-16 13:11:45 total_net_output_bytes:0
2023-08-16 13:11:45 instantaneous_input_kbps:0.00
2023-08-16 13:11:45 instantaneous_output_kbps:0.00
2023-08-16 13:11:45 rejected_connections:0
2023-08-16 13:11:45 sync_full:0
2023-08-16 13:11:45 sync_partial_ok:0
2023-08-16 13:11:45 sync_partial_err:0
2023-08-16 13:11:45 expired_keys:0
2023-08-16 13:11:45 expired_stale_perc:0.00
2023-08-16 13:11:45 expired_time_cap_reached_count:0
2023-08-16 13:11:45 expire_cycle_cpu_milliseconds:0
2023-08-16 13:11:45 evicted_keys:0
2023-08-16 13:11:45 keyspace_hits:0
2023-08-16 13:11:45 keyspace_misses:0
2023-08-16 13:11:45 pubsub_channels:0
2023-08-16 13:11:45 pubsub_patterns:0
2023-08-16 13:11:45 latest_fork_usec:0
2023-08-16 13:11:45 total_forks:0
2023-08-16 13:11:45 migrate_cached_sockets:0
2023-08-16 13:11:45 slave_expires_tracked_keys:0
2023-08-16 13:11:45 active_defrag_hits:0
2023-08-16 13:11:45 active_defrag_misses:0
2023-08-16 13:11:45 active_defrag_key_hits:0
2023-08-16 13:11:45 active_defrag_key_misses:0
2023-08-16 13:11:45 tracking_total_keys:0
2023-08-16 13:11:45 tracking_total_items:0
2023-08-16 13:11:45 tracking_total_prefixes:0
2023-08-16 13:11:45 unexpected_error_replies:0
2023-08-16 13:11:45 total_error_replies:0
2023-08-16 13:11:45 dump_payload_sanitizations:0
2023-08-16 13:11:45 total_reads_processed:0
2023-08-16 13:11:45 total_writes_processed:0
2023-08-16 13:11:45 instantaneous_lock_contention:1
2023-08-16 13:11:45 avg_lock_contention:0.000000
2023-08-16 13:11:45 storage_provider_read_hits:0
2023-08-16 13:11:45 storage_provider_read_misses:0
2023-08-16 13:11:45 
2023-08-16 13:11:45 # Replication
2023-08-16 13:11:45 role:master
2023-08-16 13:11:45 connected_slaves:0
2023-08-16 13:11:45 master_failover_state:no-failover
2023-08-16 13:11:45 master_replid:5adb45097727b0544c5e5277622cad7e40b688be
2023-08-16 13:11:45 master_replid2:0000000000000000000000000000000000000000
2023-08-16 13:11:45 master_repl_offset:0
2023-08-16 13:11:45 second_repl_offset:-1
2023-08-16 13:11:45 repl_backlog_active:0
2023-08-16 13:11:45 repl_backlog_size:1048576
2023-08-16 13:11:45 repl_backlog_first_byte_offset:0
2023-08-16 13:11:45 repl_backlog_histlen:0
2023-08-16 13:11:45 
2023-08-16 13:11:45 # CPU
2023-08-16 13:11:45 used_cpu_sys:0.141183
2023-08-16 13:11:45 used_cpu_user:0.131200
2023-08-16 13:11:45 used_cpu_sys_children:0.012834
2023-08-16 13:11:45 used_cpu_user_children:0.003993
2023-08-16 13:11:45 server_threads:1
2023-08-16 13:11:45 long_lock_waits:0
2023-08-16 13:11:45 used_cpu_sys_main_thread:0.133821
2023-08-16 13:11:45 used_cpu_user_main_thread:0.126778
2023-08-16 13:11:45 
2023-08-16 13:11:45 # Modules
2023-08-16 13:11:45 module:name=search,ver=20801,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors]
2023-08-16 13:11:45 
2023-08-16 13:11:45 # Commandstats
2023-08-16 13:11:45 cmdstat_info:calls=1,usec=21,usec_per_call=21.00,rejected_calls=0,failed_calls=0
2023-08-16 13:11:45 cmdstat_config:calls=1,usec=25,usec_per_call=25.00,rejected_calls=0,failed_calls=0
2023-08-16 13:11:45 
2023-08-16 13:11:45 # Errorstats
2023-08-16 13:11:45 
2023-08-16 13:11:45 # Cluster
2023-08-16 13:11:45 cluster_enabled:0
2023-08-16 13:11:45 
2023-08-16 13:11:45 # Keyspace
2023-08-16 13:11:45 
2023-08-16 13:11:45 # KeyDB
2023-08-16 13:11:45 mvcc_depth:0
2023-08-16 13:11:45 
2023-08-16 13:11:45 ------ CLIENT LIST OUTPUT ------
2023-08-16 13:11:45 
2023-08-16 13:11:45 ------ MODULES INFO OUTPUT ------
2023-08-16 13:11:45 # search_version
2023-08-16 13:11:45 search_version:2.8.1
2023-08-16 13:11:45 search_redis_version:255.255.255 - oss
2023-08-16 13:11:45 
2023-08-16 13:11:45 # search_index
2023-08-16 13:11:45 search_number_of_indexes:0
2023-08-16 13:11:45 
2023-08-16 13:11:45 # search_fields_statistics
2023-08-16 13:11:45 
2023-08-16 13:11:45 # search_dialect_statistics
2023-08-16 13:11:45 search_dialect_1:0
2023-08-16 13:11:45 search_dialect_2:0
2023-08-16 13:11:45 search_dialect_3:0
2023-08-16 13:11:45 
2023-08-16 13:11:45 # search_runtime_configurations
2023-08-16 13:11:45 search_concurrent_mode:OFF
2023-08-16 13:11:45 search_enableGC:ON
2023-08-16 13:11:45 search_minimal_term_prefix:2
2023-08-16 13:11:45 search_maximal_prefix_expansions:200
2023-08-16 13:11:45 search_query_timeout_ms:500
2023-08-16 13:11:45 search_timeout_policy:return
2023-08-16 13:11:45 search_cursor_read_size:1000
2023-08-16 13:11:45 search_cursor_max_idle_time:300000
2023-08-16 13:11:45 search_max_doc_table_size:1000000
2023-08-16 13:11:45 search_max_search_results:1000000
2023-08-16 13:11:45 search_max_aggregate_results:-1
2023-08-16 13:11:45 search_search_pool_size:20
2023-08-16 13:11:45 search_index_pool_size:8
2023-08-16 13:11:45 search_gc_scan_size:100
2023-08-16 13:11:45 search_min_phonetic_term_length:3
2023-08-16 13:11:45 
2023-08-16 13:11:45 ------ FAST MEMORY TEST ------
2023-08-16 13:11:45 *** Preparing to test memory region 55d7f6679000 (2334720 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae66e68000 (8388608 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae67669000 (8388608 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae68379000 (28672 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae68380000 (11010048 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae68ffc000 (45056 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae6903a000 (32768 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae691b8000 (4096 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae691f5000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae69253000 (4096 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae6938c000 (4096 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae69399000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae69573000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae6958f000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae695d7000 (4096 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae696b5000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae69980000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae69b12000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae69b79000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae69c69000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae69d5c000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae6a029000 (16384 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae6a04c000 (16384 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae6a06b000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae6a39b000 (12288 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae6a670000 (16384 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae6a729000 (8192 bytes)
2023-08-16 13:11:45 *** Preparing to test memory region 7fae6a75d000 (4096 bytes)
2023-08-16 13:11:46 .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
2023-08-16 13:11:46 Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
2023-08-16 13:11:46 
2023-08-16 13:11:46 ------ DUMPING CODE AROUND EIP ------
2023-08-16 13:11:46 Symbol: (null) (base: (nil))
2023-08-16 13:11:46 Module: /etc/libs/redisearch.so (base 0x7fae67e69000)
2023-08-16 13:11:46 $ xxd -r -p /tmp/dump.hex /tmp/dump.bin
2023-08-16 13:11:46 $ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
2023-08-16 13:11:46 ------
2023-08-16 13:11:46 
2023-08-16 13:11:46 === KEYDB BUG REPORT END. Make sure to include from START to END. ===
2023-08-16 13:11:46 
2023-08-16 13:11:46        Please report the crash by opening an issue on github:
2023-08-16 13:11:46 
2023-08-16 13:11:46            https://github.com/JohnSully/KeyDB/issues
2023-08-16 13:11:46 
2023-08-16 13:11:46   Suspect RAM error? Use keydb-server --test-memory to verify it.
2023-08-16 13:11:46 
2023-08-16 13:12:09 1:1:C 16 Aug 2023 18:12:09.751 * Notice: "active-replica yes" implies "replica-read-only no"
2023-08-16 13:12:09 1:1:C 16 Aug 2023 18:12:09.751 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
2023-08-16 13:12:09 1:1:C 16 Aug 2023 18:12:09.751 # KeyDB version=255.255.255, bits=64, commit=441b74a9, modified=1, pid=1, just started
2023-08-16 13:12:09 1:1:C 16 Aug 2023 18:12:09.751 # Configuration loaded
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.755 * monotonic clock: POSIX clock_gettime
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.755 * Running mode=standalone, port=6379.
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.756 # Server initialized
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.761 * <search> Redis version found by RedisSearch : 255.255.255 - oss
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.761 * <search> RediSearch version 2.8.1 (Git=HEAD-251cc55f)
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.761 * <search> Low level api version 1 initialized successfully
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.761 * <search> concurrent writes: OFF, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results:  1000000, search pool size: 20, index pool size: 8, 
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.766 * <search> Initialized thread pools!
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.766 * <search> Enabled diskless replication
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.766 * <search> Enabled role change notification
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.766 * Module 'search' loaded from /etc/libs/redisearch.so
2023-08-16 13:12:09 
2023-08-16 13:12:09 
2023-08-16 13:12:09 === KEYDB BUG REPORT START: Cut & paste starting from here ===
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.766 # KeyDB 255.255.255 crashed by signal: 11, si_code: 1
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.766 # Accessing address: 0x48
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.766 # Crashed running the instruction at: 0x7fc618c108e6
2023-08-16 13:12:09 
2023-08-16 13:12:09 ------ STACK TRACE ------
2023-08-16 13:12:09 EIP:
2023-08-16 13:12:09 /etc/libs/redisearch.so(+0x1a78e6) [0x7fc618c108e6]
2023-08-16 13:12:09 
2023-08-16 13:12:09 Backtrace:
2023-08-16 13:12:09 /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fc61ac15420]
2023-08-16 13:12:09 /etc/libs/redisearch.so(+0x1a78e6) [0x7fc618c108e6]
2023-08-16 13:12:09 keydb-server *:6379(+0x26aef5) [0x562c060c1ef5]
2023-08-16 13:12:09 keydb-server *:6379(InitServerLast()+0x6e) [0x562c061958de]
2023-08-16 13:12:09 keydb-server *:6379(main+0x566) [0x562c06062096]
2023-08-16 13:12:09 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fc61aa33083]
2023-08-16 13:12:09 keydb-server *:6379(_start+0x2e) [0x562c0608767e]
2023-08-16 13:12:09 
2023-08-16 13:12:09 ------ REGISTERS ------
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # 
2023-08-16 13:12:09 RAX:0000000000000000 RBX:0000000000000003
2023-08-16 13:12:09 RCX:0000000000000003 RDX:0000000000000001
2023-08-16 13:12:09 RDI:00007ffe7c65f6d0 RSI:0000000000000003
2023-08-16 13:12:09 RBP:00007ffe7c65f6d0 RSP:00007ffe7c65f630
2023-08-16 13:12:09 R8 :0000000000000000 R9 :0000000000001000
2023-08-16 13:12:09 R10:0000562c06703b5d R11:0000000000000246
2023-08-16 13:12:09 R12:00007fc618f7f098 R13:0000562c068618f8
2023-08-16 13:12:09 R14:00007ffe7c65f6d0 R15:00007fc619694820
2023-08-16 13:12:09 RIP:00007fc618c108e6 EFL:0000000000010246
2023-08-16 13:12:09 CSGSFS:002b000000000033
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f63f) -> 0000000000000000
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f63e) -> 0000000000000000
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f63d) -> 0000000000000000
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f63c) -> 00007fc618fd2a80
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f63b) -> 00007fc6196684e0
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f63a) -> 0000562c060d5be0
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f639) -> 0000562c060c571d
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f638) -> 00007fc618fc3400
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f637) -> 00007fc600000000
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f636) -> 0000000000000000
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f635) -> 0100562c00000003
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f634) -> 00000000196807a1
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f633) -> 0000562c060c1ef5
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f632) -> 0000000000000000
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f631) -> 00007ffe7c65f660
2023-08-16 13:12:09 1:1:M 16 Aug 2023 18:12:09.767 # (00007ffe7c65f630) -> 0000000000000003
2023-08-16 13:12:09 
2023-08-16 13:12:09 ------ INFO OUTPUT ------
2023-08-16 13:12:09 # Server
2023-08-16 13:12:09 redis_version:255.255.255
2023-08-16 13:12:09 redis_git_sha1:441b74a9
2023-08-16 13:12:09 redis_git_dirty:1
2023-08-16 13:12:09 redis_build_id:82759f545caa9a17
2023-08-16 13:12:09 redis_mode:standalone
2023-08-16 13:12:09 os:Linux 5.15.49-linuxkit-pr x86_64
2023-08-16 13:12:09 arch_bits:64
2023-08-16 13:12:09 multiplexing_api:epoll
2023-08-16 13:12:09 atomicvar_api:atomic-builtin
2023-08-16 13:12:09 gcc_version:9.4.0
2023-08-16 13:12:09 process_id:1
2023-08-16 13:12:09 process_supervised:no
2023-08-16 13:12:09 run_id:60cefe6e49af2dc1eacd60dd2a73791e637b5d1a
2023-08-16 13:12:09 tcp_port:6379
2023-08-16 13:12:09 server_time_usec:1692209529749462
2023-08-16 13:12:09 uptime_in_seconds:0
2023-08-16 13:12:09 uptime_in_days:0
2023-08-16 13:12:09 hz:10
2023-08-16 13:12:09 configured_hz:10
2023-08-16 13:12:09 lru_clock:14487929
2023-08-16 13:12:09 executable:/data/keydb-server
2023-08-16 13:12:09 config_file:/etc/keydb.conf
2023-08-16 13:12:09 availability_zone:
2023-08-16 13:12:09 
2023-08-16 13:12:09 # Clients
2023-08-16 13:12:09 connected_clients:0
2023-08-16 13:12:09 cluster_connections:0
2023-08-16 13:12:09 maxclients:10000
2023-08-16 13:12:09 client_recent_max_input_buffer:0
2023-08-16 13:12:09 client_recent_max_output_buffer:0
2023-08-16 13:12:09 blocked_clients:0
2023-08-16 13:12:09 tracking_clients:0
2023-08-16 13:12:09 clients_in_timeout_table:0
2023-08-16 13:12:09 current_client_thread:0
2023-08-16 13:12:09 thread_0_clients:0
2023-08-16 13:12:09 
2023-08-16 13:12:09 # Memory
2023-08-16 13:12:09 used_memory:1743832
2023-08-16 13:12:09 used_memory_human:1.66M
2023-08-16 13:12:09 used_memory_rss:0
2023-08-16 13:12:09 used_memory_rss_human:0B
2023-08-16 13:12:09 used_memory_peak:1743832
2023-08-16 13:12:09 used_memory_peak_human:1.66M
2023-08-16 13:12:09 used_memory_peak_perc:100.28%
2023-08-16 13:12:09 used_memory_overhead:8
2023-08-16 13:12:09 used_memory_startup:0
2023-08-16 13:12:09 used_memory_dataset:1743824
2023-08-16 13:12:09 used_memory_dataset_perc:100.00%
2023-08-16 13:12:09 allocator_allocated:0
2023-08-16 13:12:09 allocator_active:0
2023-08-16 13:12:09 allocator_resident:0
2023-08-16 13:12:09 total_system_memory:8346099712
2023-08-16 13:12:09 total_system_memory_human:7.77G
2023-08-16 13:12:09 used_memory_lua:37888
2023-08-16 13:12:09 used_memory_lua_human:37.00K
2023-08-16 13:12:09 used_memory_scripts:0
2023-08-16 13:12:09 used_memory_scripts_human:0B
2023-08-16 13:12:09 number_of_cached_scripts:0
2023-08-16 13:12:09 maxmemory:0
2023-08-16 13:12:09 maxmemory_human:0B
2023-08-16 13:12:09 maxmemory_policy:noeviction
2023-08-16 13:12:09 allocator_frag_ratio:-nan
2023-08-16 13:12:09 allocator_frag_bytes:0
2023-08-16 13:12:09 allocator_rss_ratio:-nan
2023-08-16 13:12:09 allocator_rss_bytes:0
2023-08-16 13:12:09 rss_overhead_ratio:-nan
2023-08-16 13:12:09 rss_overhead_bytes:0
2023-08-16 13:12:09 mem_fragmentation_ratio:-nan
2023-08-16 13:12:09 mem_fragmentation_bytes:0
2023-08-16 13:12:09 mem_not_counted_for_evict:4
2023-08-16 13:12:09 mem_replication_backlog:0
2023-08-16 13:12:09 mem_clients_slaves:0
2023-08-16 13:12:09 mem_clients_normal:0
2023-08-16 13:12:09 mem_aof_buffer:8
2023-08-16 13:12:09 mem_allocator:jemalloc-5.2.1
2023-08-16 13:12:09 active_defrag_running:0
2023-08-16 13:12:09 lazyfree_pending_objects:0
2023-08-16 13:12:09 lazyfreed_objects:0
2023-08-16 13:12:09 storage_provider:none
2023-08-16 13:12:09 available_system_memory:unavailable
2023-08-16 13:12:09 
2023-08-16 13:12:09 # Persistence
2023-08-16 13:12:09 loading:0
2023-08-16 13:12:09 current_cow_size:0
2023-08-16 13:12:09 current_cow_size_age:0
2023-08-16 13:12:09 current_fork_perc:0.00
2023-08-16 13:12:09 current_save_keys_processed:0
2023-08-16 13:12:09 current_save_keys_total:0
2023-08-16 13:12:09 rdb_changes_since_last_save:0
2023-08-16 13:12:09 rdb_bgsave_in_progress:0
2023-08-16 13:12:09 rdb_last_save_time:1692209529
2023-08-16 13:12:09 rdb_last_bgsave_status:ok
2023-08-16 13:12:09 rdb_last_bgsave_time_sec:-1
2023-08-16 13:12:09 rdb_current_bgsave_time_sec:-1
2023-08-16 13:12:09 rdb_last_cow_size:0
2023-08-16 13:12:09 aof_enabled:1
2023-08-16 13:12:09 aof_rewrite_in_progress:0
2023-08-16 13:12:09 aof_rewrite_scheduled:0
2023-08-16 13:12:09 aof_last_rewrite_time_sec:-1
2023-08-16 13:12:09 aof_current_rewrite_time_sec:-1
2023-08-16 13:12:09 aof_last_bgrewrite_status:ok
2023-08-16 13:12:09 aof_last_write_status:ok
2023-08-16 13:12:09 aof_last_cow_size:0
2023-08-16 13:12:09 module_fork_in_progress:0
2023-08-16 13:12:09 module_fork_last_cow_size:0
2023-08-16 13:12:09 aof_current_size:0
2023-08-16 13:12:09 aof_base_size:0
2023-08-16 13:12:09 aof_pending_rewrite:0
2023-08-16 13:12:09 aof_buffer_length:0
2023-08-16 13:12:09 aof_rewrite_buffer_length:0
2023-08-16 13:12:09 aof_pending_bio_fsync:0
2023-08-16 13:12:09 aof_delayed_fsync:0
2023-08-16 13:12:09 
2023-08-16 13:12:09 # Stats
2023-08-16 13:12:09 total_connections_received:0
2023-08-16 13:12:09 total_commands_processed:2
2023-08-16 13:12:09 instantaneous_ops_per_sec:0
2023-08-16 13:12:09 total_net_input_bytes:0
2023-08-16 13:12:09 total_net_output_bytes:0
2023-08-16 13:12:09 instantaneous_input_kbps:0.00
2023-08-16 13:12:09 instantaneous_output_kbps:0.00
2023-08-16 13:12:09 rejected_connections:0
2023-08-16 13:12:09 sync_full:0
2023-08-16 13:12:09 sync_partial_ok:0
2023-08-16 13:12:09 sync_partial_err:0
2023-08-16 13:12:09 expired_keys:0
2023-08-16 13:12:09 expired_stale_perc:0.00
2023-08-16 13:12:09 expired_time_cap_reached_count:0
2023-08-16 13:12:09 expire_cycle_cpu_milliseconds:0
2023-08-16 13:12:09 evicted_keys:0
2023-08-16 13:12:09 keyspace_hits:0
2023-08-16 13:12:09 keyspace_misses:0
2023-08-16 13:12:09 pubsub_channels:0
2023-08-16 13:12:09 pubsub_patterns:0
2023-08-16 13:12:09 latest_fork_usec:0
2023-08-16 13:12:09 total_forks:0
2023-08-16 13:12:09 migrate_cached_sockets:0
2023-08-16 13:12:09 slave_expires_tracked_keys:0
2023-08-16 13:12:09 active_defrag_hits:0
2023-08-16 13:12:09 active_defrag_misses:0
2023-08-16 13:12:09 active_defrag_key_hits:0
2023-08-16 13:12:09 active_defrag_key_misses:0
2023-08-16 13:12:09 tracking_total_keys:0
2023-08-16 13:12:09 tracking_total_items:0
2023-08-16 13:12:09 tracking_total_prefixes:0
2023-08-16 13:12:09 unexpected_error_replies:0
2023-08-16 13:12:09 total_error_replies:0
2023-08-16 13:12:09 dump_payload_sanitizations:0
2023-08-16 13:12:09 total_reads_processed:0
2023-08-16 13:12:09 total_writes_processed:0
2023-08-16 13:12:09 instantaneous_lock_contention:1
2023-08-16 13:12:09 avg_lock_contention:0.000000
2023-08-16 13:12:09 storage_provider_read_hits:0
2023-08-16 13:12:09 storage_provider_read_misses:0
2023-08-16 13:12:09 
2023-08-16 13:12:09 # Replication
2023-08-16 13:12:09 role:master
2023-08-16 13:12:09 connected_slaves:0
2023-08-16 13:12:09 master_failover_state:no-failover
2023-08-16 13:12:09 master_replid:430ccf03402583463508f73e514dbe3eefa57cb0
2023-08-16 13:12:09 master_replid2:0000000000000000000000000000000000000000
2023-08-16 13:12:09 master_repl_offset:0
2023-08-16 13:12:09 second_repl_offset:-1
2023-08-16 13:12:09 repl_backlog_active:0
2023-08-16 13:12:09 repl_backlog_size:1048576
2023-08-16 13:12:09 repl_backlog_first_byte_offset:0
2023-08-16 13:12:09 repl_backlog_histlen:0
2023-08-16 13:12:09 
2023-08-16 13:12:09 # CPU
2023-08-16 13:12:09 used_cpu_sys:0.043222
2023-08-16 13:12:09 used_cpu_user:0.038900
2023-08-16 13:12:09 used_cpu_sys_children:0.004934
2023-08-16 13:12:09 used_cpu_user_children:0.000000
2023-08-16 13:12:09 server_threads:1
2023-08-16 13:12:09 long_lock_waits:0
2023-08-16 13:12:09 used_cpu_sys_main_thread:0.038756
2023-08-16 13:12:09 used_cpu_user_main_thread:0.035775
2023-08-16 13:12:09 
2023-08-16 13:12:09 # Modules
2023-08-16 13:12:09 module:name=search,ver=20801,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors]
2023-08-16 13:12:09 
2023-08-16 13:12:09 # Commandstats
2023-08-16 13:12:09 cmdstat_config:calls=1,usec=34,usec_per_call=34.00,rejected_calls=0,failed_calls=0
2023-08-16 13:12:09 cmdstat_info:calls=1,usec=19,usec_per_call=19.00,rejected_calls=0,failed_calls=0
2023-08-16 13:12:09 
2023-08-16 13:12:09 # Errorstats
2023-08-16 13:12:09 
2023-08-16 13:12:09 # Cluster
2023-08-16 13:12:09 cluster_enabled:0
2023-08-16 13:12:09 
2023-08-16 13:12:09 # Keyspace
2023-08-16 13:12:09 
2023-08-16 13:12:09 # KeyDB
2023-08-16 13:12:09 mvcc_depth:0
2023-08-16 13:12:09 
2023-08-16 13:12:09 ------ CLIENT LIST OUTPUT ------
2023-08-16 13:12:09 
2023-08-16 13:12:09 ------ MODULES INFO OUTPUT ------
2023-08-16 13:12:09 # search_version
2023-08-16 13:12:09 search_version:2.8.1
2023-08-16 13:12:09 search_redis_version:255.255.255 - oss
2023-08-16 13:12:09 
2023-08-16 13:12:09 # search_index
2023-08-16 13:12:09 search_number_of_indexes:0
2023-08-16 13:12:09 
2023-08-16 13:12:09 # search_fields_statistics
2023-08-16 13:12:09 
2023-08-16 13:12:09 # search_dialect_statistics
2023-08-16 13:12:09 search_dialect_1:0
2023-08-16 13:12:09 search_dialect_2:0
2023-08-16 13:12:09 search_dialect_3:0
2023-08-16 13:12:09 
2023-08-16 13:12:09 # search_runtime_configurations
2023-08-16 13:12:09 search_concurrent_mode:OFF
2023-08-16 13:12:09 search_enableGC:ON
2023-08-16 13:12:09 search_minimal_term_prefix:2
2023-08-16 13:12:09 search_maximal_prefix_expansions:200
2023-08-16 13:12:09 search_query_timeout_ms:500
2023-08-16 13:12:09 search_timeout_policy:return
2023-08-16 13:12:09 search_cursor_read_size:1000
2023-08-16 13:12:09 search_cursor_max_idle_time:300000
2023-08-16 13:12:09 search_max_doc_table_size:1000000
2023-08-16 13:12:09 search_max_search_results:1000000
2023-08-16 13:12:09 search_max_aggregate_results:-1
2023-08-16 13:12:09 search_search_pool_size:20
2023-08-16 13:12:09 search_index_pool_size:8
2023-08-16 13:12:09 search_gc_scan_size:100
2023-08-16 13:12:09 search_min_phonetic_term_length:3
2023-08-16 13:12:09 
2023-08-16 13:12:09 ------ FAST MEMORY TEST ------
2023-08-16 13:12:09 *** Preparing to test memory region 562c06849000 (2334720 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc617a68000 (8388608 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc618269000 (8388608 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc618f79000 (28672 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc618f80000 (11010048 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc619bd0000 (45056 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc619c0e000 (32768 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc619d8c000 (4096 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc619dc9000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc619e27000 (4096 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc619f60000 (4096 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc619f6d000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61a147000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61a163000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61a1ab000 (4096 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61a289000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61a554000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61a6e6000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61a74d000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61a83d000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61a930000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61abfd000 (16384 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61ac20000 (16384 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61ac3f000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61af6f000 (12288 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61b244000 (16384 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61b2fd000 (8192 bytes)
2023-08-16 13:12:09 *** Preparing to test memory region 7fc61b331000 (4096 bytes)
2023-08-16 13:12:09 .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
2023-08-16 13:12:09 Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
2023-08-16 13:12:09 
2023-08-16 13:12:09 ------ DUMPING CODE AROUND EIP ------
2023-08-16 13:12:09 Symbol: (null) (base: (nil))
2023-08-16 13:12:09 Module: /etc/libs/redisearch.so (base 0x7fc618a69000)
2023-08-16 13:12:09 $ xxd -r -p /tmp/dump.hex /tmp/dump.bin
2023-08-16 13:12:09 $ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
2023-08-16 13:12:09 ------
2023-08-16 13:12:09 
2023-08-16 13:12:09 === KEYDB BUG REPORT END. Make sure to include from START to END. ===
2023-08-16 13:12:09 
2023-08-16 13:12:09        Please report the crash by opening an issue on github:
2023-08-16 13:12:09 
2023-08-16 13:12:09            https://github.com/JohnSully/KeyDB/issues
2023-08-16 13:12:09 
2023-08-16 13:12:09   Suspect RAM error? Use keydb-server --test-memory to verify it.
2023-08-16 13:12:09

Dockerfile

FROM ubuntu:20.04 AS builder 

WORKDIR /keydb
RUN apt-get update
RUN apt-get install -y git make python3 build-essential make
RUN git clone -b v2.8.1 --recursive https://github.com/RediSearch/RediSearch.git
RUN cd RediSearch && make setup  && make build 

FROM eqalpha/keydb:latest AS prod 
ADD keydb.conf /etc/keydb.conf
COPY --from=builder /keydb/RediSearch/bin/linux-x64-release/search/redisearch.so /etc/libs/redisearch.so
CMD ["keydb-server", "/etc/keydb.conf"]

keydb.conf

dir /data/ 
protected-mode no 
appendonly yes

save 900 1
save 300 10
save 60 10000

active-replica yes
replica-read-only no
multi-master yes
replica-serve-stale-data yes

maxmemory-policy noeviction

loadmodule /etc/libs/redisearch.so

Tried again this time removed the loadmodule /etc/libs/redisearch.so line from keydb.conf and ran MODULE LOAD /etc/libs/redisearch.so in RedisInsight which worked though then running FT.CREATE yields "RediSearch is not available for this database" and then details and buttons to get started for free

Did it work for you @sapvs when loading redisearch later?

[sapvs]

@CanRau , Yes it works for me when loading redisearch later. I am loading them lazily, using wrapper entrypoint script, which loads both modules after keydb has started.

Redis JSON has no issues loading at startup, only RediSearch module causes the crash.

Setup repo with same here.
Github Repo: sapvs/keydb_json_search
Docker Images: vsaps/keydb_jsonsearch

root@46154a85d68f:/data# keydb-cli
127.0.0.1:6379> FT.CREATE idx:movie ON hash PREFIX 1 "movie:" SCHEMA title TEXT SORTABLE release_year NUMERIC SORTABLE rating NUMERIC SORTABLE genre TAG SORTABLE
OK

Crash issue on load with RediSearch still exists, and also KeyDB image created from main branch shows redis_version:255.255.255

[msotheeswaran-sc]

Hi @sapvs and @CanRau Snap's legal team reviewed the various Redis module licenses and determined that we cannot support any of them, including fixing bugs related to them, so I cannot help here.

[CanRau]

As long as the project is not abandoned that's fine with me, though curious what about Redis 7

[msotheeswaran-sc]

Unfortunately we don't have a timeline for when we will merge Redis 7, the team is mostly busy with internal work for Snap.

[sotnikovse]

Having the same problem with redisearch in keydb v6.3.4, rejson v2.6.0, redisearch v2.8.1
Tried different versions of redisearch 2.0, 2.2, 2.4 etc, but no luck, the behavior is the same
Lazy loading of the redisearch module works, but on server reload, persistence .rdb is not load indexes created by FT.CREATE
What can be done in this case?
Is there any possibility to reload (manually load) the .rdb file after lazy load of the redisearch module?
Or do I have to recreate the indexes every server reload?
I just started learning keydb and redis, I will be glad for any help

[msotheeswaran-sc]

@sotnikovse there is a "debug reload" command that will reload the rdb that you can try

[sotnikovse]

"debug reload" with loaded redisearch module failed with the same error. And with flash enabled, recreating of FT.CREATE works unexpectedly.
Ok, it doesn't seem like a good idea to use redisearch.

[keithchew]

Hi @sotnikovse

Good to see you have reached the same point as me in January, which I posted the issue here:

#531

Like you, I got redisearch working with flash enabled, but unfortunately, the indexes are not created upon server restart. When I investigated further, I observed that with flash enabled, there is no "loading" of an RDB on startup, as the data resides on disk. This means the RDB loading is not performed, rather keydb accesses data directly from disk on demand.

My next step was to see if I can create the search indexes on startup, but did not get far. We basically need to read all data from disk on startup and "index" them in the redisearch module. If anyone ever gets a chance to look into this, please let me know, keen to see redisearch working on keydb with flash.

[keithchew]

Hi @sotnikovse @sapvs @CanRau

Looks like there have been 2 version upgrades since I tested in January. I can confirm the this PR introduced in v6.3.3 is the cause of the crash in redisearch.so:

#536

The line that causes the crash is:

moduleFireServerEvent(REDISMODULE_EVENT_LOADING, REDISMODULE_SUBEVENT_LOADING_ENDED, NULL);

Unfortunately, to fix this, you will need to patch redisearch in 2 places:

src/redismodule.h

#define REDISMODULE_SUBEVENT_LOADING_FLASH_START 6

and src/spec.c

  if (subevent == REDISMODULE_SUBEVENT_LOADING_RDB_START ||
      subevent == REDISMODULE_SUBEVENT_LOADING_AOF_START ||
      subevent == REDISMODULE_SUBEVENT_LOADING_FLASH_START || <----- ADD THIS
      subevent == REDISMODULE_SUBEVENT_LOADING_REPL_START) {

After this, you will not get the crash on startup.

[sotnikovse]

Hi @keithchew, thank you very much for this solution, keydb v6.3.4, rejson v2.6.0 and patched redisearch v2.8.1 startup without crash.
Yes, with flash enabled the indexes are not created on restart.

[keithchew]

hi @sotnikovse not a problem, it is my pleasure. Please note that if you start with an empty flash folder and a non-empty RDB (eg backup from another instance), the above solution is not complete/robust. This is because the current code will fire both RDB_START and FLASH_START events, which is not ideal when you add code to create/load indexes at startup. You will need to update the startup process slightly to accommodate this scenario.