[BUG] Inability to write when reached maxmemory in FLASH mode

[jianjun126]

Describe the bug
Hi,
I have tested in centos7 with memtier_benchmark (v1.4.0). When the memory reaches its limit, there may be instances of inability to write or very low write performance, such as:

After the memory limit is reached, if key timeout or starting test queries performance, the write performance will also degrade to a very low value. such as:

To reproduce

keydb command:
./keydb-server ./keydb.conf --storage-provider flash /data1/6333/ --storage-provider-options "use_direct_reads=true;allow_mmap_reads=false;use_direct_writes=true;allow_mmap_writes=false"
keydb config:
keydb.conf.zip

memtier_benchmark command：
taskset -c 26-29,78-84 memtier_benchmark -s 127.0.0.1 -p 6333 -t 4 -c 20 -n 2000000 --distinct-client-seed --command="set __key__ __data__ ex 66000" --key-prefix="testkey_v3_" --key-minimum=100000000 --key-maximum=999000000 -R -d 800

Expected behavior
When reaching maximum memory in FLASH mode, keydb can write normally and maintain good performance

Additional information
I have tried to modify some parameters of keydb, but the above phenomenon still exists.

[paulmchen]

It sounds running out of the replication buffer hard limit and triggers a fast fullsync (check your log for confirmation). Increasing the replication hardlimit in your conf will help if it is running out of the replication hardlimit.

client-output-buffer-limit replica 2gb 2gb 60

In the SSD case, you can also adjust the following parameters to a value larger than 1, to better handle large write loads.

maxmemory-eviction-tenacity 35

[jianjun126]

@paulmchen Thank you very much for your suggestion. After I modify the configuration as your suggestion, there are still same issues with inability to write and low write performance. There are two screenshot of the test results.

[jianjun126]

@paulmchen
Hi paulmchen,
I tried to modify some other parameters, but still couldn't solve the problem. Could you please give me some more advice on this issue. Thanks for your kind attention and look forward your prompt reply.

[msotheeswaran-sc]

Hi @jianjun126 are the writes being rejected or hanging, it could be a similar problem to #646, however with the expireset taking up all the memory instead of the slots_to_keys map.

[paulmchen]

@jianjun126 as it looks like you have a single master configuration (without a slave), it won't be a replication backlog issue or a fast full sync issue. It is suggested to run FlameGraph to determine where the bottleneck is causing low CPU usage.

Follow these instructions to set up and run FlameGraph to identify your system's performance issue:

1. Place FlameGraph tool under an empty folder: (e.g. /FlameGraph).
   git clone https://github.com/brendangregg/FlameGraph # or download it from github

2. Start your KeyDB server and run your client workload

3. Before you see low or zero QPS, do the following:
   cd /FlameGraph
   perf record --call-graph=dwarf -p [process_id] # where process_id is the process id of the KeyDB server running

   record the perf for about 30 seconds (when seeing 0QPs or till the end of the slow cpu period if less than 1 minute), then stop the recording.

   perf script > out.perf # send result to out.perf file
   ./stackcollapse-perf.pl out.perf > out.folded
   ./flamegraph.pl out.folded > keydb-fg-result.svg

4. It would be helpful if you could share the svg file so we can check where the CPU boundary is

[jianjun126]

@paulmchen
Hi paulmchen,
I have gotten two cvg results with FlameGraph. It takes about 50 seconds to run perf each time. Here are two relevant screenshots of memtier_benchmark.
"LOW OPS"

"ZERO OPS"

"FlameGraph result while LOW OPS"

"FlameGraph result while ZERO OPS"

[paulmchen]

According to the FG diagram, more than 73% of CPU cycles are spent performing evictions. EvictionPoolPopulate also consumes more than 55% of CPU cycles (see evict.cpp). My suspicion is that your volatile-ttl setting for maxmemory-policy may not work well with your benchmark command with ex=66000

memtier_benchmark -s 127.0.0.1 -p 6333 -t 4 -c 20 -n 2000000 --distinct-client-seed --command="set key data ex 66000" --key-prefix="testkey_v3_" --key-minimum=100000000 --key-maximum=999000000 -R -d 800

You may try the following:

1. Run the benchmark on a separate client server. The benchmark tool is currently running on the same keydb server, and when there is heavy workload, both the client and the server consume CPU/memory resources. (bind 127.0.0.1 -::1 from the configuration is to listen for the loopback network interface. You should change it to use your private IP instead. This enables you to call the keydb server from another client server on the same subnet to run your heavy workload without competing resources between the client and the server.

2. Try with this maxmemory_policy instead:
   maxmemory-policy allkeys-lru

3. If you have more than 1 core on your keydb server, giving it a bit more threads to handle the workload will help a lot. For example:

   server-threads 4

[jianjun126]

@paulmchen
Hi paulmchen,

My test environment is a dual-socket server, where the CPU is 8269CY, and the memory has 6 channels. And almost no other tasks are executed at the same time during the test. So, it's almost certainly not a hardware resource issue.
The other two parameters you suggested, I have also tested many times. Such as server-threads 4, server-threads 5, maxmemory-policy allkeys-lru, allkeys-lfu.

If necessary, I will retest as soon as possible with your suggested parameters and environment.

[quwu0820]

铂金8269CY，26核52线程，主频2.5G；

[jianjun126]

@paulmchen @msotheeswaran

I wrote a script to test the performance of keydb when the write rate is low, but I found that even at a low write rate, there is still the problem of not being able to write.

The test method is to randomly write 5000-8000 times per second for 50-70 seconds within 300 seconds; and write 250 times per second for the rest of the 300 seconds, and the amount of data written each time is 8000 bytes.

During the test, I adjusted the maxmemory, whether it is 1GB, 4GB, or 24GB, this problem will occur. The timing of the occurrence is approximately 1.5 hours or 3.5 hours of the test duration.

[paulmchen]

That is a bug, the following commit addresses the 0 and low QPS issue. 0 QPS is caused by eviction right after the maxmemory is reached. However, it may have a side effect on the code which could cause issues after reaching maxmemory, and memory usage may continue to grow. @JohnSully, John, could the following commits be added to main as well, are there any side effects? For example, the memory will keep growing without effectively being evicted on time?

#439

@jianjun126 you can try this commit and see if it helps.

[jianjun126]

@paulmchen
Hi paulmchen,
I tried the code modified in #439 with memtier_benchmark, it does addresses the 0 OPS issue, but there was still low OPS issue.
At the same time, as guessed, the memory keep growing without effectively being evicted.