The GC process may break the CURP Durability in some extreme cases

[Phoenix500526]

Currently, our gc process mainly relies on running gc_task every GC_INTERVAL. In some extreme cases, it threatens to lose those completed-but-yet-to-sync commands.

For instance, a client sends a request to a leader and the leader executes this command in the fast path. But before syncing the command execution result to others, it crashes. And for some unknown reason, the cluster cannot reach the same page about who is the next leader. After GC_INTERVAL, this command is removed from spec_pool via the gc task. After ten seconds or so, the new leader is finally elected. But unfortunately, the speculatively executed command is gone. The new leader cannot recover it from the spec_pool.

[Phoenix500526]

Read the following docs for more information:

1. Coracle: Evaluating Consensus at the Internet Edge
2. Raft does not Guarantee Liveness in the face of Network Faults