-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: Update
snapshots-creator
readme (#1163)
## What ❔ - Updates `snapshots_creator` crate readme with more info. - A typo in the command https://github.com/matter-labs/zksync-era/tree/main/core/bin/snapshots_creator zk run snapshots_creator --> zk run snapshots-creator - .yarnrc dir appears on linux after calling zk commands. --> moved to gitignore ## Why ❔ Wrong command. + extra file ## Checklist - [x] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [x] Documentation comments have been added / updated. - [x] Spellcheck has been run via `zk spellcheck`. - [x] Linkcheck has been run via `zk linkcheck`. --------- Co-authored-by: Alex Ostrovski <aov@matterlabs.dev>
- Loading branch information
Showing
2 changed files
with
61 additions
and
15 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,6 +20,7 @@ tags | |
# Yarn files | ||
.yarn/ | ||
.yarnrc.yml | ||
.yarnrc | ||
|
||
zksync_pk.key | ||
dist | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,24 +1,69 @@ | ||
# Snapshots Creator | ||
|
||
Snapshot creator is small command line tool for creating a snapshot of zkSync node for EN node to be able to initialize | ||
to a certain L1 Batch. | ||
Snapshot creator is a command line tool for creating an app-level snapshot of the node storage at a certain L1 batch. A | ||
snapshot does not contain full transaction history, but rather [a minimal subset of information](#snapshots-format) | ||
needed to bootstrap an external node. A snapshot is always made at an L1 batch boundary. | ||
|
||
Snapshots do not contain full transactions history, but rather a minimal subset of information needed to bootstrap EN | ||
node. | ||
Compared to Postgres dumps, app-level snapshots are much more compact; as of Feb 2024, the mainnet snapshot is approx. | ||
13 GB (gzipped). | ||
|
||
Usage (local development):\ | ||
First run `zk env dev` \ | ||
then the creator can be run using: | ||
`zk run snapshots_creator` | ||
> **Warning.** Snapshot creator is in the early stage of development; expect bugs and incomplete / outdated docs. Use at | ||
> your own risk. | ||
Snapshot contents can be stored based on blob_store config either in local filesystem or GS. | ||
## Local testing | ||
|
||
Usage for local development (assuming the development environment [has been set up](../../../docs/guides/setup-dev.md)): | ||
|
||
1. Run `zk env dev` | ||
2. Generate storage logs, e.g. by running a load test with a command like | ||
`ACCOUNTS_AMOUNT=40 DURATION_SEC=60 zk run loadtest`. | ||
3. Run the creator using `zk run snapshots-creator`. | ||
|
||
To check that creating the snapshot succeeded, you may use the `snapshots` namespace of the JSON-RPC API of the main | ||
node. With the local setup, the corresponding `curl` args are as follows: | ||
|
||
```shell | ||
curl -X POST -H "Content-Type: application/json" \ | ||
--data '{"jsonrpc": "2.0", "id": 1, "method": "snapshots_getAllSnapshots", "params": [] }' \ | ||
'http://localhost:3050' | ||
# Should return an array of L1 batch numbers for all created snapshots, such as | ||
# { | ||
# "snapshotsL1BatchNumbers": [42] | ||
# } | ||
|
||
curl -X POST -H "Content-Type: application/json" \ | ||
--data '{"jsonrpc": "2.0", "id": 1, "method": "snapshots_getSnapshot", "params": [42] }' \ | ||
'http://localhost:3050' | ||
# Returns metadata for a specific snapshot containing `l1BatchNumber`, `miniblockNumber` | ||
# and other fields. | ||
``` | ||
|
||
By default, in the local setup snapshots are stored in the `artifacts/storage_logs_snapshots` directory relative to the | ||
repository root. The storage location can be configured using the object store configuration to either use the local | ||
filesystem, or Google Cloud Storage (GCS). Beware that for end-to-end testing of snapshot recovery, changes applied to | ||
the main node configuration must be reflected in the external node configuration. | ||
|
||
Creating a snapshot is a part of the [snapshot recovery integration test]. You can run the test using | ||
`yarn snapshot-recovery-test snapshot-recovery-test`. It requires the main node to be launched with a command like | ||
`zk server --components api,tree,eth,state_keeper,commitment_generator`. | ||
|
||
## Snapshots format | ||
|
||
Each snapshot consists of three types of objects (see | ||
[snapshots.rs](https://github.com/matter-labs/zksync-era/blob/main/core/lib/types/src/snapshots.rs)) : header, storage | ||
logs chunks and factory deps: | ||
Each snapshot consists of three types of data (see [`snapshots.rs`] for exact definitions): | ||
|
||
- **Header:** Includes basic information, such as the miniblock / L1 batch of the snapshot, miniblock / L1 batch | ||
timestamps, miniblock hash and L1 batch root hash. Returned by the methods in the `snapshots` namespace of the | ||
JSON-RPC API of the main node. | ||
- **Storage log chunks:** Latest values for all VM storage slots ever written to at the time the snapshot is made. | ||
Besides key–value pairs, each storage log record also contains the L1 batch number of its initial write and its | ||
enumeration index; both are used to restore the contents of the `initial_writes` table. Chunking storage logs is | ||
motivated by their parallel generation; each chunk corresponds to a distinct non-overlapping range of hashed storage | ||
keys. (This should be considered an implementation detail for the purposes of snapshot recovery; recovery must not | ||
rely on any particular key distribution among chunks.) Stored as gzipped Protobuf messages in an [object store]; each | ||
chunk is a separate object. | ||
- **Factory dependencies:** All bytecodes deployed on L2 at the time the snapshot is made. Stored as a single gzipped | ||
Protobuf message in an object store. | ||
|
||
- Snapshot Header (currently returned by snapshots namespace of JSON-RPC API) | ||
- Snapshot Storage logs chunks (most likely to be stored in gzipped protobuf files, but this part is still WIP) : | ||
- Factory dependencies (most likely to be stored as protobufs in the very near future) | ||
[`snapshots.rs`]: ../../lib/types/src/snapshots.rs | ||
[object store]: ../../lib/object_store | ||
[snapshot recovery integration test]: ../../tests/snapshot-recovery-test/tests/snapshot-recovery.test.ts |