Table of Contents
- =nil; Proof Market Toolchain
- Introduction
- Proof Systems Compatibility
- Starting with a ready Docker image
- Building the toolchain
- Proof Market Beta Access
- Proof Market Interaction
- Common issues
- Community
This repository provides a set of scripts and tools required to participate in the
=nil; Foundation's Proof Market.
There are three primary roles (parties) in the Proof Market:
- Proof requesters are applications that require zero-knowledge proofs, and make requests for them on the Proof Market.
- Proof producers are owners of computational infrastructure, who generate proofs for the requests made by proof requesters.
- Circuit developers make zero-knowledge circuits, that are used to generate requests and subsequent proofs.
The Proof Market toolchain has tools for proof requesters and producers. To learn more about the Proof Market and these roles, read the Proof Market documentation.
If you're interested in circuit development, check out the zkLLVM compiler and zkLLVM template project.
Proof Maket Toolchain is tested with the following versions of the circuit development tools:
| Tool | Version |
|---|---|
| zkLLVM | 0.0.79 |
The quickest way to start working with the Proof Market toolchain is to use the
Docker image ghcr.io/nilfoundation/proof-market-toolchain:latest.
It has all the parts of the toolchain:
- All required dependencies.
- Scripts for interaction with Proof Market API.
- The
proof-generatorbinary for building proofs.
cd /your/zk/project
docker run -it \
--volume $(pwd):/opt/project \
--user $(id -u ${USER}):$(id -g ${USER}) \
ghcr.io/nilfoundation/proof-market-toolchain:latest
# --volume mounts your project's directory into the container
# --user solves issues with file permissions
root@abc123:/proof-market-toolchain proof-producer -h
root@abc123:/proof-market-toolchain python3 scripts/prepare_statement.py --helpRemember to pull the image often to get the latest release:
docker pull ghcr.io/nilfoundation/proof-market-toolchain:latest .Clone the repo with submodules:
git clone --recurse-submodules git@github.com:NilFoundation/proof-market-toolchain.git
cd proof-market-toolchainWhen you pull newer commits, always checkout the submodules as well:
git submodule update --init --recursiveIf you have errors on cloning submodules, setup the SSH keys on GitHub and try cloning again.
Note: If you just need the latest release version, use the Docker image at
ghcr.io/nilfoundation/proof-market-toolchain:latest.
You can build a Docker image from a particular commit:
docker build -t ghcr.io/nilfoundation/proof-market-toolchain:latest .Now you can run a container based on this image:
docker run -it \
-v $(pwd):/opt/project \
--user $(id -u ${USER}):$(id -g ${USER}) \
ghcr.io/nilfoundation/proof-market-toolchain:latest
root@abc123:/proof-market-toolchain proof-producer -h
root@abc123:/proof-market-toolchain python3 scripts/prepare_statement.py --helpWhen you build the image, tag can be anything. For example, you can use the current commit's hash as the tag:
docker build -t ghcr.io/nilfoundation/proof-market-toolchain:$(git rev-parse --short HEAD) .The final image is built from a base image tagged
ghcr.io/nilfoundation/proof-market-toolchain:base.
It has precompiled Boost and all other required dependencies.
If you want to update dependencies, change them in the Dockerfile.base,
and then rebuild the base image:
docker build -t ghcr.io/nilfoundation/proof-market-toolchain:base --file Dockerfile.base .To use proof market binaries, run them from the same container.
On *nix systems, the following dependencies need to be installed:
apt install \
build-essential \
libssl-dev \
cmake \
clang-12 \
git \
autoconf \
libc-ares-dev \
libfmt-dev \
gnutls-dev \
liblz4-dev \
libprotobuf-dev \
libyaml-cpp-dev \
libhwloc-dev \
pkg-config \
xfslibs-dev \
systemtap-sdt-devInstall Boost either manually or from your distributive's repository. Please make sure you are installing the version 1.76. Follow the guide to install version 1.76 manually.
We have tested for the following set of versions of the libraries:
clang-12
clang++12
boost == 1.76
cmake >= 3.22.1
autoconf >= 2.71
automake >= 1.16.5
libc-ares-dev >= 1.18.1
libfmt-dev >= 8.1.1
liblz4-dev >= 1.9.3
gnutls-dev >= 7.81
libprotobuf-dev >= 3.12.4
libyaml-cpp-dev >= 0.2.2
libhwloc-dev >= 2.7.0
libsctp-dev >= 1.0.19
ragel >= 6.10
We are aware of compilation issues with boost > 1.76 and clang > 12.0. Please use the versions recommended above
mkdir build
cd build
cmake -G "Unix Makefiles" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=/usr/bin/clang-12 \
-DCMAKE_CXX_COMPILER=/usr/bin/clang++-12 \
..
# Single-threaded version (recommended)
cmake --build . -t proof-generatorStart with installing requirements. We recommend using virtualenv:
pip3 install --user virtualenv
python3 -m virtualenv venv
source venv/bin/activate
pip3 install -r requirements.txtPlease see the documentation
on how to get access for proof market.
Or navigate to scripts/ directory and run the following commands.
First, create a username and password.
Store them in files .user and .secret:
$ cd scripts
$ echo '<username>' > .user
$ echo '<password>' > .secretSignup to the proof market with the following command. You'll need to repeat your username and password here.
$ python3 signup.py user -u <username> -p <password> -e <e-mail>
This command will return user metadata in JSON format:
{
"user": "username",
"active": true,
"extra": {},
"error": false,
"code": 201
}If you wish to submit generated proofs to the Proof Market, you need to also register as a proof producer:
$ python3 signup.py producer --url https://example.comIt will return producer metadata in JSON format:
{
"_key": "...",
"_id": "producer/...",
"_rev": "_fneT7J2---",
"description": "Generic Producer",
"url": "https://example.com",
"name": "username",
"createdOn": 1677588821180,
"updatedOn": 1677588821180
}Below we will list a set of operations a user can follow along which demonstrates the market operation interaction between the above entities.
The easiest way to create a circuit definition for Proof Market is zkLLVM. All dependencies and build instructions are inside the zkLLVM's repository. If you wish to interact with existing circuits/statements, please go to step 3.
Circuits can be generated by any one. They are serialised & published on the proof market. This allows for reuse of the circuits by all other proof requesters. Circuits need a set of public inputs.
2.1 Create a new circuit (using zkLLVM)
make -C ${ZKLLVM_BUILD:-build} <circuit target name> -j$(nproc)
2.2 Prepare a statement with circuit description for Proof Market (using this repository)
Circuits are stored as a statement structure on Proof Market.
Statement description example can be found in example/statements/ directory
Please ensure you have selected the option to create *ll IR files in the zkLLVM setup.
python3 scripts/prepare_statement.py \
-c <zkllvm output> \
-o <statement description file> \
-n <statement name> \
-t <statement type> \
--private | --publicThe --private or --public parameters are mutually exclusive:
- With
--private, the statement will be accessible only by its ID. For experiments and development purposes, make your statements private. - With
--public, the statement will be openly listed on the Proof Market.
Provide the necessary information listed in the output statement file
2.2 Publish it to Proof Market
This statement can now be pushed to the Proof market via python script
python3 scripts/statement_tools.py push --file <json file with statement description>
You will be returned an object containing a _key filed -- unique descriptor of the statement
2.3 Get all statements
A list of all available statements can be obtained by
python3 scripts/statement_tools.py get
The proof requester can create a request order. It has additional details such as what are they willing to pay for it and public inputs.
python3 scripts/request_tools.py push --cost <cost of the request> --file <json file with public_input> --key <key of the statement>
The proof requester can check their request with
python3 scripts/request_tools.py get --key <key of the request>
Here the proof requester waits for matching engine to either match an existing order or wait for proposals to be submitted against this circuit/statement.
While the proposal is up, we now view at the marketplace from the perspective of a proof producer. In steps 2-4 , the requester put out a request. Now the producer can observe them in the marketplace and start replying with an proposal.
python3 scripts/proposal_tools.py push --cost <cost of the proposal> --key <key of the statement>
The proof producer can check their proposal with
python3 scripts/proposal_tools.py get --key <key of the proposal>
Proof Market runs a matching algorithm between requests and proposals for each new request/proposal. It chooses the cheapest proposal that fits the requirements of the proof requester.
Proposal's status 'processing' means that the proposal was matched with a request. Now it is time to generate a proof for the proof producer.
First of all, the proof producer needs circuit definition:
python3 scripts/statement_tools.py get --key <key of the statement> -o <output file>
Next, public input of the request:
python3 scripts/public_input_get.py --key <request key> -o <output file path>
Execute the below to generate a proof:
cd build
./bin/proof-generator/proof-generator --proof_out=<output file> --circuit_input=<statement from Proof Market> --public_input=<public input from Proof Market>
Readme for Proof Producer daemon in located here.
The proof generator can now submit the proof to the marketplace, where if verified, they will get the reward.
python3 scripts/proof_tools.py push --request_key <key of the request> --proposal_key <key of the proposal> --file <file with the proof>
You can provide only one of two possible keys
Now the proof requester is able to get their proof either by request key or proof key.
python3 scripts/proof_tools.py get --request_key <key of the request>
Validation of the proof is not part of the tool chain. Validation flow is implemented in the lorem ipsum cli repository here.
If you have more than one compiler installed i.e g++ & clang++. The make system might pick up the former. You can explicitly force usage of clang++ by finding the path and passing it in the variable below.
`which clang++`
cmake .. -DCMAKE_CXX_COMPILER=<path to clang++ from above>
Git maintains a few places where submodule details are cached. Sometimes updates do not come through. ex: Deletion , updating a url of a previously checked out submodule.It is advisable to check these locations for remains or try a new checkout.
- .gitmodules
- .git/config
- .git/modules/*
Please ensure you are using 1.76 version of boost as the higher versions have an incompatible API which will be updated in due course.
On macOS, these dependencies are required for compilation
fmt gnutls protobuf yaml-cpp ragel hwloc