[Feature]: Snapshot

[themanforfree]

When to create

1. When a node falls too far behind the Leader, the Leader creates a Snapshot and sends it to the corresponding follower.
2. When a user actively creates a Snapshot.

How to create

Create a Snapshot through the StorageApi's interface, which returns a Snapshot structure. The specific implementation is determined by the underlying storage engine and must implement the AsyncRead and AsyncWrite traits for reading and writing Snapshot-related files.

How to send

Snapshots are sent in two situations:

1. Leader sends to Follower - this is done through the InstallSnapshot RPC, where the curp client sends the Snapshot to the lagging Follower. The request is a stream that sends the Snapshot in chunks to the Follower.
2. User actively requests a Snapshot - this is done through the Snapshot RPC of the maintenance service, where the user sends a SnapshotRequest request and the server returns a stream that sends the Snapshot in chunks back to the user.

How to recover

When the Follower receives the InstallSnapshot RPC, it writes the Snapshot to its local storage and then applies the Snapshot to the local storage through the StorageApi's apply_snapshot interface, which is implemented by the underlying storage engine.

The Snapshot file obtained by the user is only used for disaster recovery and is recovered to obtain the business data through a separate tool.

Proto

message InstallSnapshotRequest {
    uint64 term = 1;
    uint64 leader_id = 2;
    uint64 last_included_index = 3;
    uint64 last_included_term = 4;
    uint64 offset = 5;
    bytes data = 6;
    bool done = 7;
}

message InstallSnapshotResponse {
    uint64 term = 1;
}

service Protocol {
    ...

    rpc InstallSnapshot (stream InstallSnapshotRequest) returns (InstallSnapshotResponse);

}

Abstract

trait StorageApi {
    type Error:  std::error::Error + Send + Sync + 'static;
    
    type Snapshot: Snapshot;

    ...

    fn snapshot(&self) -> Result<Self::Snapshot,Self::Error>;

    fn apply_snapshot(&self, snapshot: Self::Snapshot) -> Result<(),Self::Error>;
}

trait Snapshot: Read + Write {

    fn size(&self) -> u64;
}

Discussion

we will persist term and other metadata directly, do we need to create snapshots for them regularly? Or do we only need to regularly compact the log?

[markcty]

Bridge betweenCurp and Xline:

Curp will tell Xline to take a snapshot using command executor:

#[async_trait]
pub trait CommandExecutor<C>: Sync + Send + Clone + std::fmt::Debug
where
    C: Command,
{
    // ...

    /// Take a snapshot
    async fn snapshot(&self) -> Result<Box<dyn SnapshotApi>, Self::Error>;

    /// Install a snapshot
    async fn install_snapshot(&self, snapshot: Box<dyn SnapshotApi>) -> Result<(), Self::Error>;
}

[markcty]

And there should be Send + Sync + Seek for snapshot api