Cluster was Set Up, But Couldn't Handle 'INCR counter:1' and Crashed

[JasonLou99]

I tried to establish RedisRaft cluster with https://redis.com/blog/redisraft-new-strong-consistency-deployment-option/, 3 Nodes (1 Leader 2 Follower).
The cluster was set up, but when I tried to type INCR counter:1 in Leader, the cluster couldn't handle it and crashed.The error information is as follows:

jason@webassembly120:~/redisraft$ redis-server --port 5001 --dbfilename raft1.rdb --loadmodule ./redisraft.so --raft.log-filename raftlog1.db --raft.addr 192.168.10.120:5001 --protected-mode no
8349:C 28 Mar 2023 16:54:27.530 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
8349:C 28 Mar 2023 16:54:27.530 # Redis version=7.0.9, bits=64, commit=00000000, modified=0, pid=8349, just started
8349:C 28 Mar 2023 16:54:27.530 # Configuration loaded
8349:M 28 Mar 2023 16:54:27.531 * Increased maximum number of open files to 10032 (it was originally set to 1024).
8349:M 28 Mar 2023 16:54:27.531 * monotonic clock: POSIX clock_gettime
8349:M 28 Mar 2023 16:54:27.531 # Server initialized
8349:M 28 Mar 2023 16:54:27.531 * RedisRaft version 255.255.255 [316745c]
8349:M 28 Mar 2023 16:54:27.535 * Raft module loaded, state is 'uninitialized'
8349:M 28 Mar 2023 16:54:27.535 * Module 'raft' loaded from ./redisraft.so
8349:M 28 Mar 2023 16:54:27.535 * Ready to accept connections
8349:M 28 Mar 2023 16:54:34.113 * State change: Node is now a leader, term 1
8349:M 28 Mar 2023 16:54:34.113 * Cluster Membership: term:1 index:1 nodes: id=736509181,voting=0,active=1,addr=-
8349:M 28 Mar 2023 16:54:34.113 * Raft Cluster initialized, node id: 736509181, dbid: 0d9d72234f69dd2ac762ba34c33aeaf5

=== REDIS BUG REPORT START: Cut & paste starting from here ===
8349:M 28 Mar 2023 16:56:22.725 # Redis 7.0.9 crashed by signal: 11, si_code: 1
8349:M 28 Mar 2023 16:56:22.725 # Accessing address: 0x24
8349:M 28 Mar 2023 16:56:22.725 # Crashed running the instruction at: 0x55f7112323b4

------ STACK TRACE ------
EIP:
redis-server *:5001(callReplyIsResp3+0x4)[0x55f7112323b4]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f9214e19420]
redis-server *:5001(callReplyIsResp3+0x4)[0x55f7112323b4]
redis-server *:5001(RM_ReplyWithCallReply+0x98)[0x55f711202688]
./redisraft.so(RaftExecuteCommandArray+0x1e0)[0x7f9214a3c800]
./redisraft.so(+0x267c9)[0x7f9214a3e7c9]
./redisraft.so(raft_apply_entry+0x7f)[0x7f9214a4bcbf]
./redisraft.so(raft_apply_all+0x61)[0x7f9214a4bda1]
./redisraft.so(raft_exec_operations+0x6d)[0x7f9214a4d42d]
./redisraft.so(raft_flush+0x3b)[0x7f9214a4d66b]
./redisraft.so(handleBeforeSleep+0x41)[0x7f9214a3f0f1]
redis-server *:5001(moduleFireServerEvent+0x135)[0x55f7112088c5]
redis-server *:5001(beforeSleep+0x18e)[0x55f71115fe8e]
redis-server *:5001(aeProcessEvents+0xf8)[0x55f71115b8c8]
redis-server *:5001(aeMain+0x1d)[0x55f71115bd7d]
redis-server *:5001(main+0x33a)[0x55f71115781a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f9214c35083]
redis-server *:5001(_start+0x2e)[0x55f711157f3e]

------ REGISTERS ------
8349:M 28 Mar 2023 16:56:22.726 #
RAX:00007f921471f600 RBX:00007f9214738838
RCX:0000020009000000 RDX:0000000000000000
RDI:0000000000000000 RSI:0000000000000000
RBP:00007f921471f600 RSP:00007fff3966ba28
R8 :00007f9200000000 R9 :0000000000000038
R10:00007f9214a72748 R11:00007f9214603280
R12:0000000000000000 R13:00007f9214629500
R14:00007f92146630a3 R15:0000000000000000
RIP:000055f7112323b4 EFL:0000000000010246
CSGSFS:002b000000000033
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba37) -> 0000000000000004
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba36) -> 000000000000000d
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba35) -> 000000013966bbf8
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba34) -> 00007f9214738800
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba33) -> 00007f9214a73858
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba32) -> 00007fff3966baa0
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba31) -> 0000000000000000
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba30) -> 0000000012f66d40
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba2f) -> 00007f9214a72760
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba2e) -> 00007f9214a3c800
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba2d) -> 00007f920dc00000
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba2c) -> 0000000000000000
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba2b) -> 00007f9214629500
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba2a) -> b6812890495e0300
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba29) -> 00007f9214a5c5a7
8349:M 28 Mar 2023 16:56:22.726 # (00007fff3966ba28) -> 000055f711202688

------ INFO OUTPUT ------

Server

redis_version:7.0.9
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:448f00a942bc1f77
redis_mode:standalone
os:Linux 5.15.0-67-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:9.4.0
process_id:8349
process_supervised:no
run_id:bca2f3412f3d1f2865e7f1644c61a2cf14503388
tcp_port:5001
server_time_usec:1679993782724042
uptime_in_seconds:115
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:2272182
executable:/home/jason/redisraft/redis-server
config_file:
io_threads_active:0

Clients

connected_clients:3
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:8
client_recent_max_output_buffer:0
blocked_clients:1
tracking_clients:0
clients_in_timeout_table:0

Memory

used_memory:1892968
used_memory_human:1.81M
used_memory_rss:11272192
used_memory_rss_human:10.75M
used_memory_peak:2051928
used_memory_peak_human:1.96M
used_memory_peak_perc:92.25%
used_memory_overhead:1722080
used_memory_startup:1716496
used_memory_dataset:170888
used_memory_dataset_perc:96.84%
allocator_allocated:2042608
allocator_active:2531328
allocator_resident:6098944
total_system_memory:8300429312
total_system_memory_human:7.73G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.24
allocator_frag_bytes:488720
allocator_rss_ratio:2.41
allocator_rss_bytes:3567616
rss_overhead_ratio:1.85
rss_overhead_bytes:5173248
mem_fragmentation_ratio:6.09
mem_fragmentation_bytes:9420224
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:5400
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

Persistence

loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1679993667
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

Stats

total_connections_received:6
total_commands_processed:15
instantaneous_ops_per_sec:0
total_net_input_bytes:343
total_net_output_bytes:172889
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:1
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:12
total_writes_processed:10
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:3
reply_buffer_expands:0

Replication

role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:a27035db5ae447b58462633a8c342b95161b07d9
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:0.142058
used_cpu_user:0.358955
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.142198
used_cpu_user_main_thread:0.355496

Modules

module:name=raft,ver=1,api=1,filters=1,usedby=[],using=[],options=[]

Commandstats

cmdstat_raft.node:calls=2,usec=66,usec_per_call=33.00,rejected_calls=0,failed_calls=0
cmdstat_raft:calls=1,usec=31,usec_per_call=31.00,rejected_calls=0,failed_calls=0
cmdstat_raft.cluster:calls=1,usec=3047,usec_per_call=3047.00,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=3,usec=4,usec_per_call=1.33,rejected_calls=0,failed_calls=0
cmdstat_config|set:calls=1,usec=7,usec_per_call=7.00,rejected_calls=0,failed_calls=0
cmdstat_command:calls=2,usec=976,usec_per_call=488.00,rejected_calls=0,failed_calls=0
cmdstat_command|docs:calls=1,usec=586,usec_per_call=586.00,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=4,usec=11,usec_per_call=2.75,rejected_calls=0,failed_calls=0

Errorstats

Latencystats

latency_percentiles_usec_raft.node:p50=16.063,p99=17.023,p99.9=17.023
latency_percentiles_usec_raft.cluster:p50=3047.423,p99=3047.423,p99.9=3047.423
latency_percentiles_usec_config|get:p50=1.003,p99=2.007,p99.9=2.007
latency_percentiles_usec_config|set:p50=7.007,p99=7.007,p99.9=7.007
latency_percentiles_usec_command:p50=462.847,p99=516.095,p99.9=516.095
latency_percentiles_usec_command|docs:p50=589.823,p99=589.823,p99.9=589.823
latency_percentiles_usec_auth:p50=3.007,p99=3.007,p99.9=3.007

Cluster

cluster_enabled:0

Keyspace

------ CLIENT LIST OUTPUT ------
id=10 addr=192.168.10.121:46378 laddr=192.168.10.120:5001 fd=11 name= age=92 idle=92 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1800 events=r cmd=auth user=default redir=-1 resp=2
id=12 addr=192.168.10.122:52464 laddr=192.168.10.120:5001 fd=13 name= age=79 idle=79 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1800 events=r cmd=auth user=default redir=-1 resp=2
id=13 addr=127.0.0.1:46448 laddr=127.0.0.1:5001 fd=14 name= age=48 idle=0 flags=b db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=13 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22309 events=r cmd=raft user=default redir=-1 resp=2

------ MODULES INFO OUTPUT ------

raft_version

raft_version:255.255.255
raft_git_sha1:316745c1

raft_general

raft_dbid:0d9d72234f69dd2ac762ba34c33aeaf5
raft_node_id:736509181
raft_state:up
raft_role:leader
raft_is_voting:yes
raft_voted_for:-1
raft_leader_id:736509181
raft_current_term:1
raft_num_nodes:3
raft_num_voting_nodes:3
raft_node1:id=855214545,state=connected,voting=yes,addr=192.168.10.121,port=5001,last_conn_secs=92,conn_errors=0,conn_oks=1
raft_node2:id=1133110375,state=connected,voting=yes,addr=192.168.10.122,port=5001,last_conn_secs=79,conn_errors=0,conn_oks=1

raft_log

raft_log_entries:7
raft_current_index:7
raft_commit_index:7
raft_last_applied_index:6
raft_file_size:1994
raft_cache_memory_size:1749
raft_cache_entries:7
raft_client_attached_entries:0
raft_fsync_count:7
raft_fsync_max_microseconds:939
raft_fsync_avg_microseconds:713

raft_snapshot

raft_snapshot_filename:raft1.rdb
raft_snapshot_last_idx:0
raft_snapshot_last_term:0
raft_snapshot_size:0
raft_snapshot_time_secs:-1
raft_snapshots_created:0
raft_snapshots_received:0
raft_snapshot_in_progress:no
raft_snapshot_in_progress_last_idx:-1
raft_snapshot_in_progress_last_term:-1

raft_clients

raft_proxy_reqs:0
raft_proxy_failed_reqs:0
raft_proxy_failed_responses:0
raft_proxy_outstanding_reqs:0

raft_stats

raft_appendreq_received:0
raft_appendreq_with_entry_received:0
raft_snapshotreq_received:0
raft_exec_throttled:0
raft_num_sessions:0

------ CONFIG DEBUG OUTPUT ------
lazyfree-lazy-server-del no
lazyfree-lazy-expire no
io-threads-do-reads no
lazyfree-lazy-eviction no
activedefrag no
repl-diskless-sync yes
lazyfree-lazy-user-del no
sanitize-dump-payload no
proto-max-bulk-len 512mb
lazyfree-lazy-user-flush no
slave-read-only yes
repl-diskless-load disabled
client-query-buffer-limit 1gb
list-compress-depth 0
replica-read-only yes
io-threads 1

------ FAST MEMORY TEST ------
8349:M 28 Mar 2023 16:56:22.727 # Bio thread for job type #0 terminated
8349:M 28 Mar 2023 16:56:22.727 # Bio thread for job type #1 terminated
8349:M 28 Mar 2023 16:56:22.727 # Bio thread for job type #2 terminated
*** Preparing to test memory region 55f7113bd000 (2306048 bytes)
*** Preparing to test memory region 55f712f35000 (270336 bytes)
*** Preparing to test memory region 7f91ff800000 (8388608 bytes)
*** Preparing to test memory region 7f9200000000 (135168 bytes)
*** Preparing to test memory region 7f9204000000 (135168 bytes)
*** Preparing to test memory region 7f9208000000 (135168 bytes)
*** Preparing to test memory region 7f920c000000 (4194304 bytes)
*** Preparing to test memory region 7f920c600000 (8388608 bytes)
*** Preparing to test memory region 7f920ce00000 (4194304 bytes)
*** Preparing to test memory region 7f920d400000 (8388608 bytes)
*** Preparing to test memory region 7f920dc00000 (4194304 bytes)
*** Preparing to test memory region 7f920e199000 (8388608 bytes)
*** Preparing to test memory region 7f920e99a000 (8388608 bytes)
*** Preparing to test memory region 7f920f19b000 (8388608 bytes)
*** Preparing to test memory region 7f920f99c000 (8388608 bytes)
*** Preparing to test memory region 7f921019d000 (8388608 bytes)
*** Preparing to test memory region 7f921099e000 (8388608 bytes)
*** Preparing to test memory region 7f921119f000 (8388608 bytes)
*** Preparing to test memory region 7f92119a0000 (8388608 bytes)
*** Preparing to test memory region 7f92121a1000 (8388608 bytes)
*** Preparing to test memory region 7f92129a2000 (8388608 bytes)
*** Preparing to test memory region 7f92131a2000 (2621440 bytes)
*** Preparing to test memory region 7f9214200000 (8388608 bytes)
*** Preparing to test memory region 7f9214a71000 (36864 bytes)
*** Preparing to test memory region 7f9214a9d000 (8192 bytes)
*** Preparing to test memory region 7f9214dff000 (24576 bytes)
*** Preparing to test memory region 7f9214e24000 (16384 bytes)
*** Preparing to test memory region 7f92150fb000 (16384 bytes)
*** Preparing to test memory region 7f9215240000 (4096 bytes)
*** Preparing to test memory region 7f9215396000 (8192 bytes)
*** Preparing to test memory region 7f92153da000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: callReplyIsResp3 (base: 0x55f7112323b0)
Module: redis-server *:5001 (base 0x55f7110e9000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x55f7112323b0 -D -b binary -m i386:x86-64 /tmp/dump.bin

8349:M 28 Mar 2023 16:56:23.043 # dump of function (hexdump of 132 bytes):
f30f1efa8b472483e004c30f1f440000f30f1efa488b4740c30f1f8000000000f30f1efa41544989d4554889f5534889fbbf50000000e825f7f3ffc740240100000048895808488958100fb67bff89f983e10780f9040f87bb4cf2ff488d35d5290d000fb6c94863148e4801f23effe28b7bf7488978184c892048c74048000000004889

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

   Please report the crash by opening an issue on github:

       http://github.com/redis/redis/issues

If a Redis module was involved, please open in the module's repo instead.

Suspect RAM error? Use redis-server --test-memory to verify it.

Some other issues could be detected by redis-server --check-system
Segmentation fault (core dumped)

[tezc]

@JasonLou99 Currently, RedisRaft requires Redis build from "unstable" branch.
You can follow instructions here: https://github.com/RedisLabs/redisraft#creating-a-redisraft-cluster

RedisRaft is still being developed and as part of this effort, we push some changes to Redis. So, currently RedisRaft depends on Redis unstable branch.

Meanwhile, I'll try to make our version mismatch more strict to warn about this.