ADR-040 Implementation: Support state sync

[i-norden]

• PR: feat: ADR-040: add state sync for v2 store #10794

Child of #9816

We need to support the state sync snapshotting process for bootstrapping a node with fast sync. This will follow the current approach but will require us to offer up snapshot plans that allow the consumer to reconstruct both the SC and SS.

• Provider- offer a new snapshot plan
• Consumer- restore the snapshot
• Need to be able to reconstruct the state store and the current SMT

Proposal:
https://hackmd.io/dIt8ZaILTZSFpyqas6kTEw

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[i-norden]

Implementing ABCI snapshots for new Tendermint DB backends

Approach

We will attempt to make all needed changes conform to the same interface used in the Snapshot/Restore methods implemented by rootmulti.Store. This is the only type that will need to be serialized for snapshots, hence the only type to implement the Snapshotter interface. Adapting this pattern for the new backends will mostly be a matter of defining the serialization units for export/import.

The most straightforward way to do this is to just define a new SnapshotItem message type encapsulating a KV-pair, and stream all entries from a store in this format. This will allow the SDK stores to be completely agnostic to the backend used, while the existing export/import procedure will remain mostly the same.

• Both new backends also provide native dump (or "backup") features. These could potentially be used in implementing ABCI snapshots for some performance benefit, but this would force the boostrapping node to use the same backend for the same stores as every peer node, since the data would be in a backend-specific format. So, this option can probably be ruled out.

Interface

We can define new generic interface types to export and import encoded messages, and implement these for all relevant persistent stores. These will reflect the current IAVL export types, but deal directly with SnapshotItems (or another generic type). The existing logic can be refactored accordingly.

type PortableStore interface {
    Export(uint64) (Exporter, error)
    Import(uint64) (Importer, error)
}

type Exporter interface {
    Next() (SnapshotItem, error)
    Close()
}

type Importer interface {
    Add(SnapshotItem)
    Commit() error
    Close()
}

Implementation

BadgerDB

The NewTransactionAt provides a view of the DB at a specific version. In "managed" mode, the version (timestamp) can be an arbitrary integer.

• CommitAt also allows restoring state at a past version, although we seem to have no use case for this as of yet.

RocksDB

RocksDB snapshots are an ephemeral view of the DB at a specific point in time. This is enough to create a snapshot directly after a commit.

• If we want to support creation from a past block height, that can be implemented with persistent copy-on-write checkpoints (which will be needed for versioned queries as well; see below).

Encoding

We define the following new protobuf message and add it as a subtype of SnapshotItem:

message SnapshotKVItem {
    bytes key = 1;
    bytes value = 2;
}

message SnapshotItem {
  oneof item {
    ...
    SnapshotKVItem kv = 3;
  }
}

The Exporter/Importer implementations for BadgerDB and RocksDB both stream instances of SnapshotKVItem. The IAVL handling code can also be wrapped so that the multistore code only has to deal with this generic interface.

[roysc]

Completed by #10794