Skip to content

Commit

Permalink
feat: New prover documentation (#2466)
Browse files Browse the repository at this point in the history
New prover documentation.

[Rendered
form](https://github.com/matter-labs/zksync-era/tree/popzxc-new-prover-docs/prover/docs/README.md)
is recommended, as it's easier to view.

Additionally, does the following:

- Introduces ordering for advanced guides (it was a mix, not it's more
or less sorted and suggests an order for reading).
- Fixes a few things in the `setup-dev.md`.
- Creates a `prover-local` config profile, so that no manual
interactions with configs are needed.
- Gets rid of `prover/setup.sh` as it's no longer needed that much.
  • Loading branch information
popzxc authored Jul 26, 2024
1 parent 7c336b1 commit 1b61d07
Show file tree
Hide file tree
Showing 29 changed files with 742 additions and 483 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ Cargo.lock
!/etc/env/configs/stage-proofs.toml
!/etc/env/configs/testnet.toml
!/etc/env/configs/mainnet.toml
!/etc/env/configs/prover-local.toml
/etc/env/l1-inits
!/etc/env/l1-inits/stage.env
!/etc/env/l1-inits/stage_proofs.env
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,6 @@

## zkEVM clarifier

[Back to ToC](../../specs/README.md)

The ZKsync zkEVM plays a fundamentally different role in the zkStack than the EVM does in Ethereum. The EVM is used to
execute code in Ethereum's state transition function. This STF needs a client to implement and run it. Ethereum has a
multi-client philosophy, there are multiple clients, and they are written in Go, Rust, and other traditional programming
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
# Deeper Overview

[Back to ToC](../../../README.md)
# Proof System Deeper Overview

The purpose of this section is to explain our new proof system from an engineering standpoint. We will examine the code
examples and how the libraries communicate.
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
28 changes: 28 additions & 0 deletions docs/guides/advanced/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# ZKsync advanced guides

This section contains more advanced guides that aim to explain complex internals of ZKsync ecosystem in an easy to grasp
way.

## Table of Contents

- [Local initialization](./01_initialization.md)
- [Deposits](./02_deposits.md)
- [Withdrawals](./03_withdrawals.md)
- [Contracts](./04_contracts.md)
- [Calls](./05_how_call_works.md)
- [Transactions](./06_how_transaction_works.md)
- [Fee model](./07_fee_model.md)
- [L2 messaging](./08_how_l2_messaging_works.md)
- [Pubdata](./09_pubdata.md)
- [Pubdata with blobs](./10_pubdata_with_blobs.md)
- [Bytecode compression](./11_compression.md)
- [EraVM intro](./12_alternative_vm_intro.md)
- [ZK intuition](./13_zk_intuition.md)
- [ZK deeper overview](./14_zk_deeper_overview.md)
- [Prover keys](./15_prover_keys.md)

Additionally, there are a few articles that cover specific topics that may be useful for developers actively working on
`zksync-era` repo.

- [Advanced debugging](./90_advanced_debugging.md)
- [Docker and CI](./91_docker_and_ci.md)
109 changes: 40 additions & 69 deletions docs/guides/setup-dev.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,42 +2,67 @@

## TL;DR

If you run on 'clean' Debian on GCP:
This is a shorter version of setup guide to make it easier subsequent initializations. If it's the first time you're
initializing the workspace, it's recommended that you read the whole guide below, as it provides more context and tips.

If you run on 'clean' Ubuntu on GCP:

```bash
# For VMs only! They don't have SSH keys, so we override SSH with HTTPS
git config --global url."https://github.com/".insteadOf git@github.com:
git config --global url."https://".insteadOf git://

# Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# NVM
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash
# All necessary stuff
sudo apt-get install build-essential pkg-config cmake clang lldb lld libssl-dev postgresql
# Docker
sudo usermod -aG docker YOUR_USER
sudo apt-get update
sudo apt-get install build-essential pkg-config cmake clang lldb lld libssl-dev postgresql apt-transport-https ca-certificates curl software-properties-common
# Install docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
sudo apt install docker-ce
sudo usermod -aG docker ${USER}

# Stop default postgres (as we'll use the docker one)
sudo systemctl stop postgresql
sudo systemctl disable postgresql
# Start docker.
sudo systemctl start docker

## You might need to re-connect (due to usermod change).

# Node & yarn
nvm install 20
# Important: there will be a note in the output to load
# new paths in your local session, either run it or reload the terminal.
npm install -g yarn
yarn set version 1.22.19

# For running unit tests
cargo install cargo-nextest
# SQL tools
cargo install sqlx-cli --version 0.7.4
# Stop default postgres (as we'll use the docker one)
sudo systemctl stop postgresql
# Start docker.
sudo systemctl start docker

# Foundry
curl -L https://foundry.paradigm.xyz | bash
foundryup --branch master
# You will need to reload your `*rc` file here

# Clone the repo to the desired location
git clone git@github.com:matter-labs/zksync-era.git
cd zksync-era
git submodule update --init --recursive
```

Don't forget to [add env variables](#Environment) and look at [tips](#tips).

## Supported operating systems

ZKsync currently can be launched on any \*nix operating system (e.g. any linux distribution or MacOS).

If you're using Windows, then make sure to use WSL 2, since WSL 1 is known to cause troubles.
If you're using Windows, then make sure to use WSL 2.

Additionally, if you are going to use WSL 2, make sure that your project is located in the _linux filesystem_, since
accessing NTFS partitions from within WSL is very slow.
Expand Down Expand Up @@ -90,38 +115,9 @@ If logging out does not resolve the issue, restarting the computer should.

## `Node` & `Yarn`

1. Install `Node` (requires version `v18.18.0`). Since our team attempts to always use the latest LTS version of
`Node.js`, we suggest you to install [nvm](https://github.com/nvm-sh/nvm). It will allow you to update `Node.js`
version easily in the future (by running `nvm use v18.18.0` in the root of the repository)
2. Install `yarn` (make sure to get version 1.22.19 - you can change the version by running `yarn set version 1.22.19`).
Instructions can be found on the [official site](https://classic.yarnpkg.com/en/docs/install/). Check if `yarn` is
installed by running `yarn -v`. If you face any problems when installing `yarn`, it might be the case that your
package manager installed the wrong package.Make sure to thoroughly follow the instructions above on the official
website. It contains a lot of troubleshooting guides in it.

## `Axel`

Install `axel` for downloading keys:

On mac:

```bash
brew install axel
```

On debian-based linux:

```bash
sudo apt-get install axel
```

Check the version of `axel` with the following command:

```
axel --version
```

Make sure the version is higher than `2.17.10`.
1. Install `Node` (requires version `v20`). The recommended way is via [nvm](https://github.com/nvm-sh/nvm).
2. Install `yarn`. Can be done via `npm install -g yarn`. Make sure to get version 1.22.19 - you can change the version
by running `yarn set version 1.22.19`.

## `clang`

Expand Down Expand Up @@ -221,33 +217,9 @@ SQLx is a Rust library we use to interact with Postgres, and its CLI is used to
features of the library.

```bash
cargo install --locked sqlx-cli --version 0.7.3
cargo install --locked sqlx-cli --version 0.7.4
```

## Solidity compiler `solc`

Install the latest solidity compiler.

On mac:

```bash
brew install solidity
```

On debian-based linux:

```bash
sudo add-apt-repository ppa:ethereum/ethereum
sudo apt-get update
sudo apt-get install solc
```

Alternatively, download a [precompiled version](https://github.com/ethereum/solc-bin) and add it to your PATH.

## Python

Most environments will have this preinstalled but if not, install Python.

## Easier method using `nix`

Nix is a tool that can fetch _exactly_ the right dependencies specified via hashes. The current config is Linux-only but
Expand All @@ -274,11 +246,10 @@ Edit the lines below and add them to your shell profile file (e.g. `~/.bash_prof
export ZKSYNC_HOME=/path/to/zksync

export PATH=$ZKSYNC_HOME/bin:$PATH

# If you're like me, uncomment:
# cd $ZKSYNC_HOME
```

## Tips

### Tip: `mold`

Optionally, you may want to optimize the build time with the modern linker, [`mold`](https://github.com/rui314/mold).
Expand Down
2 changes: 1 addition & 1 deletion etc/env/base/fri_prover.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[fri_prover]
setup_data_path = "vk_setup_data_generator_server_fri/data"
setup_data_path = "crates/bin/vk_setup_data_generator_server_fri/data"
prometheus_port = 3315
max_attempts = 10
generation_timeout_in_secs = 600
Expand Down
8 changes: 8 additions & 0 deletions etc/env/configs/prover-local.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Config for running prover locally
__imports__ = [ "base", "l1-inits/.init.env", "l2-inits/dev.init.env" ]

[eth_sender.sender]
proof_sending_mode = "OnlyRealProofs"

[fri_prover]
cloud_type = "Local"
41 changes: 8 additions & 33 deletions prover/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,36 +2,11 @@

This directory contains all the libraries and binaries related to proving of the blocks.

Directories with 'fri' suffix, are mostly used with the new proof system (Boojum).

## Components

### vk_setup_data_generator_server_fri

Set of tools to create setup keys, verification keys and verification key hashes for the circuits.

Usually run once, and then we use their outputs in multiple places in the system.

### prover_fri_gateway

Communication module between the 'main' server running the state keeper, and the proving subsystem.

### witness_generator

Creating prover jobs and saving necessary artifacts.

### prover_fri

This directory contains the main 'prover'. It can be run in two modes: either as CPU or as GPU. (controlled via 'gpu'
feature flag).

### witness_vector_generator

Only used in GPU proving mode. Prepares all the witness data using CPU, and then streams it to the prover_fri.

This is mostly used for resource efficiency (as machines with GPUs are more expensive, it allows us to run many
witness_vector_generators, that can 'share' as single gpu based prover_fri).

### proof_fri_compressor

Used as a 'last step' to compress/wrap the final FRI proof into a SNARK (to make L1 verification cheaper).
## Documentation

- [Intro](00_intro.md)
- [Setting up a GCP VM](01_gcp_vm.md)
- [Workspace setup](02_setup.md)
- [Running prover subsystem](03_launch.md)
- [Proof generation flow](04_flow.md)
- [Further reading](99_further_reading.md)
Loading

0 comments on commit 1b61d07

Please sign in to comment.