[dr-autosync] v6.5.4 QPS drop to 0 during switch sync_recovery to sync

[mayjiang0203]

Bug Report

What version of TiKV are you using?

[2023/08/19 15:39:12.244 +08:00] [INFO] [lib.rs:85] ["Welcome to TiKV"]
[2023/08/19 15:39:12.244 +08:00] [INFO] [lib.rs:90] ["Release Version:   6.5.4"]
[2023/08/19 15:39:12.244 +08:00] [INFO] [lib.rs:90] ["Edition:           Community"]
[2023/08/19 15:39:12.244 +08:00] [INFO] [lib.rs:90] ["Git Commit Hash:   e033d337f373481ada457ebd7537f568283ef42b"]
[2023/08/19 15:39:12.244 +08:00] [INFO] [lib.rs:90] ["Git Commit Branch: heads/refs/tags/v6.5.4"]
[2023/08/19 15:39:12.244 +08:00] [INFO] [lib.rs:90] ["UTC Build Time:    Unknown (env var does not exist when building)"]
[2023/08/19 15:39:12.244 +08:00] [INFO] [lib.rs:90] ["Rust Version:      rustc 1.67.0-nightly (96ddd32c4 2022-11-14)"]
[2023/08/19 15:39:12.244 +08:00] [INFO] [lib.rs:90] ["Enable Features:   pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure"]

What operating system and CPU are you using?

Steps to reproduce

1. Inject network partition during primary dc and backup dc for 1 hour;
2. Delete the chaos of network partition in step 1;
3. After about 15min, QPS drop more than 30%.

[2023/08/19 17:15:43.432 +08:00] [INFO] [cluster.go:520] ["network partition will be inject"] [source="[tiup,tikv1,tikv2,tikv3,tidb1,tidb3,pd1,pd2,benchtoolset,br,workload-ecasdb]"] [target="[pd3,tikv4,tikv5,tikv6,tidb2,tidb4]"]
[2023/08/19 18:15:43.518 +08:00] [INFO] [cluster.go:907] ["Delete chaos tiup &{e2e-dr-auto-sync-func-thin-tps-2070499-1-384 network-partition-qkhwnhaa 0xc0039dc8c0}"]

What did you expect?

No QPS drop, or drop less than 30%.

What did happened?

[mayjiang0203]

/assign @nolouch
/severity critical

[mayjiang0203]

By reducing workload pressure and set tikv config "{raftstore: {"store-io-pool-size: 2","apply-pool-size: 4"}}" , didn't hit this again.
/remove-severity critical
/severity major

[nolouch]

need to adjust the config, closed.