How to recover in writing timeout

[lv-stupidboy]

Please check the FAQ documentation before raising an issue

Describe the bug (required)

Your Environments (required)

• OS: uname -a
• Compiler: g++ --version or clang++ --version
• CPU: lscpu
• Commit id (e.g. a3ffc7d8)

How To Reproduce(required)

Steps to reproduce the behavior:
版本号3.2.1版本，且space是3副本，机器配置104C 300+内存，cpu和内存未观测到存在瓶颈，4块800G SSD磁盘，数据量20G左右

1. 通过flink-connector进行数据写入，batch size =100，写入一段时间后graph日志显示RPC超时：
   StorageClientBase-inl.h.ext: Request to ip:9779 time out : TTransportException: Timed out
   There some RPC errors: RPC failure in storageClient with without :: TTransportException: time out
   InsertVerticesExecutor failed, error E_PRC_FAILURE, part 1
   InsertVerticesExecutor failed, error E_PRC_FAILURE, part 2
   InsertVerticesExecutor failed, error E_PRC_FAILURE, part 3
2. 查询对应的storage日志：
   RaftPart.cpp:1033 Replicating log timed out : replicateLogLatencyUs 10001168
   RaftPart.cpp:1033 Replicating log timed out : replicateLogLatencyUs 10000230
   RaftPart.cpp:1033 Replicating log timed out : replicateLogLatencyUs 10001245
   RaftPart.cpp:1033 Replicating log timed out : replicateLogLatencyUs 10001037
   RaftPart.cpp:1033 Replicating log timed out : replicateLogLatencyUs 10001223
   .........
3. 如上storage日志持续打印7个小时且未恢复正常，节点处于offline状态一直未恢复

Expected behavior
1、想请问下上述情况发生可能存在哪些原因
2、节点应该如何恢复
3、单个节点offline，再提交任务为何还是写入失败，其他2个副本均正常

Additional context

[tangyuanzhang]

数据量较多的时候，rocksdb发生了compaction leader write rocksdb的时候 wirte stall 了，你可以搜下rocksdb的log日志，Stalling关键字，阻塞了可执行线程，导致了rpc超时，同时会发生选举，等compation完成以后应该就没问题了

[tangyuanzhang]

或者可以调整下执行线程数，大于你hosts的leader应该也可以解决

[lv-stupidboy]

或者可以调整下执行线程数，大于你hosts的leader应该也可以解决

感谢答复，请问下大于hosts的leader这个怎么理解。 另外针对rockesdb的一些参数配置，有最佳实践或者一些建议吗，我们再生产环境中使用发现，存在的稳定性问题比较多