Gas Benchmarks

This repository contains scripts to run benchmarks across multiple clients. Follow the instructions below to run the benchmarks locally.

Prerequisites

Make sure you have the following installed on your system:

• Python 3.10
• Docker
• Docker Compose
• .NET 8.0.x
• make (for running make commands)
• git lfs, zip (additional tools)

Setup

1. Clone the repository:

git clone https://github.com/nethermindeth/gas-benchmarks.git
cd gas-benchmarks

2. Install Python dependencies:

pip install -r requirements.txt

2.1.Install additional tools:

git lfs install
git lfs pull
sudo apt install zip

3. Prepare Kute dependencies (specific to Nethermind):

make prepare_tools

4. Create a results directory:

mkdir -p results

Running the Benchmarks

Script: Run all

For running the whole pipeline, you can use the run.sh script.

bash run.sh -t "testsPath" -w "warmupFilePath" -c "client1,client2" -r runNumber -i "image1,image2"

Example run:

run.sh -t "eest_tests/" -w "warmup-tests" -c "nethermind,geth,reth" -r 8

Flags:

• --t it's used to define the path where the tests are located.
• --w it's used to define the path where the warmup file is located.
• --c it's used to define the clients that you want to run the benchmarks. Separate the clients with a comma.
• --r it's used to define the number of iterations that you want to run the benchmarks. It's a numeric value.
• --i it's used to define the images that you want to use to run the benchmarks. Separate the images with a comma, and match the clients. Use default if you want to ignore the values.

Now you're ready to run the benchmarks locally!

Populating the PostgreSQL Database with Benchmark Data

After running benchmarks and generating report files, you can populate a PostgreSQL database with the results for further analysis. This process involves two main scripts: generate_postgres_schema.py to set up the database table, and fill_postgres_db.py to load the data.

1. Setting up the Database Schema

The generate_postgres_schema.py script creates the necessary table in your PostgreSQL database to store the benchmark data.

Usage:

python generate_postgres_schema.py \
    --db-host <your_db_host> \
    --db-port <your_db_port> \
    --db-user <your_db_user> \
    --db-name <your_db_name> \
    --table-name <target_table_name> \
    --log-level <DEBUG|INFO|WARNING|ERROR|CRITICAL>

• You will be prompted to enter the password for the specified database user.
• --table-name: Defaults to benchmark_data.
• --log-level: Defaults to INFO.

Example:

python generate_postgres_schema.py \
    --db-host localhost \
    --db-port 5432 \
    --db-user myuser \
    --db-name benchmarks \
    --table-name gas_benchmark_results

This will create a table named gas_benchmark_results (if it doesn't already exist) in the benchmarks database.

2. Populating the Database with Benchmark Data

Once the schema is set up, use fill_postgres_db.py to parse the benchmark report files (generated by run.sh or other means) and insert the data into the PostgreSQL table.

Usage:

python fill_postgres_db.py \
    --reports-dir <path_to_reports_directory> \
    --db-host <your_db_host> \
    --db-port <your_db_port> \
    --db-user <your_db_user> \
    --db-password <your_db_password> \
    --db-name <your_db_name> \
    --table-name <target_table_name> \
    --log-level <DEBUG|INFO|WARNING|ERROR|CRITICAL>

• --reports-dir: Path to the directory containing the benchmark output files (e.g., output_*.csv, raw_results_*.csv, and index.html or computer_specs.txt).
• --db-password: The password for the database user.
• --table-name: Should match the table name used with generate_postgres_schema.py. Defaults to benchmark_data.
• --log-level: Defaults to INFO.

Example:

python fill_postgres_db.py \
    --reports-dir ./results/my_benchmark_run_01 \
    --db-host localhost \
    --db-port 5432 \
    --db-user myuser \
    --db-password "securepassword123" \
    --db-name benchmarks \
    --table-name gas_benchmark_results

This script will scan the specified reports directory, parse the client benchmark data and computer specifications, and insert individual run records into the gas_benchmark_results table.

Continuous Metrics Posting

A new script run_and_post_metrics.sh is available to run the benchmarks and post metrics continuously in an infinite loop. This script updates the local repository with git pull, runs the benchmark tests, populates the PostgreSQL database, and cleans up the reports directory.

Usage:

./run_and_post_metrics.sh --table-name gas_limit_benchmarks --db-user nethermind --db-host perfnet.core.nethermind.dev --db-password "MyPass" [--warmup warmup/warmup-1000bl-16wi-24tx.txt]

Parameters:

• --table-name: The database table name where benchmark data will be inserted.
• --db-user: The database user.
• --db-host: The database host.
• --db-password: The database password.
• --warmup: (Optional) The warmup file to use. Defaults to warmup/warmup-1000bl-16wi-24tx.txt.

Examples:

Using the default warmup file:

./run_and_post_metrics.sh --table-name gas_limit_benchmarks --db-user nethermind --db-host perfnet.core.nethermind.dev --db-password "MyPass"

Using a custom warmup file and run in background:

nohup ./run_and_post_metrics.sh --table-name gas_limit_benchmarks --db-user nethermind --db-host perfnet.core.nethermind.dev --db-password "MyPass" --warmup "warmup/custom_warmup.txt" &

Prevent creating of nohup.txt (to save disk space):

nohup ./run_and_post_metrics.sh --table-name gas_limit_benchmarks --db-user nethermind --db-host perfnet.core.nethermind.dev --db-password "MyPass" --warmup "warmup/custom_warmup.txt" > /dev/null 2>&1 &

Integration with EELS

Run the tests generated by the EELS framework

By default, gas-benchmarks replays Execution Layer Spec (EELS) payloads with the Kute runner.

You can use the EELS repository to produce new benchmark definitions and run them with the following steps:

1. Place a genesis file inside scripts/genesisfiles/YOUR_CLIENT, for example:
   • scripts/genesisfiles/geth/chainspec.json
   • scripts/genesisfiles/geth/zkevmgenesis.json
2. Capture EELS tests (writes payload files into eest_tests/ by default):

python3 capture_eest_tests.py -o eest_tests -x 1M,worst_bytecode

• The capture script downloads the latest benchmark@v* release from the execution-specs repository. Ensure your new tests are included in a published benchmark release (or pass --release-tag benchmark@vX.Y.Z) before attempting to capture them.

3. Generate warmup payloads that match the captured tests:

python3 make_warmup_tests.py -s eest_tests/ -d warmup-tests -g scripts/genesisfiles/geth/zkevmgenesis.json

4. Execute the tests: perform step 4 from "Running the Benchmarks" using eest_tests/ as the test path and the warmup file generated in step 3:

bash run.sh -t "eest_tests/" -w "warmup-tests" -c "client1,client2" -r runNumber -i "image1,image2"

All documentation now references eest_tests/ as the canonical dataset; legacy tests/ or tests-vm/ folders are no longer shipped.

To author new EELS benchmarks, follow the guidance in the execution-specs repository under tests/benchmark/.

Execute the tests directly from the EELS repository

1. Clone the EELS repository:

git clone https://github.com/ethereum/execution-specs.git
cd execution-specs

2. Create a virtual environment and install the dependencies:

uv sync --all-extras

3. Use execute remote to run the tests:

uv run execute remote -v --fork=Prague --rpc-seed-key=ACCOUNT --rpc-chain-id=1 --rpc-endpoint=http://127.0.0.1:8545 tests -- -m benchmark -n 1

Note: you will need an account with some ETH (native tokens) to run the tests. Specify the private key using --rpc-seed-key. Also, --rpc-endpoint should point to the node you want to test against (it may be a remote node).

EELS stateful tests generator

The EELS stateful generator creates deterministic, reproducible execution payloads by running Execution Layer Spec tests against a local Nethermind node through a proxy. It consists of two cooperating tools:

• eest_stateful_generator.py - Main orchestrator that boots a Nethermind node from snapshot, runs EELS through a proxy, and captures payloads
• mitm_addon.py - mitmproxy addon that intercepts transactions and produces engine payloads grouped by test metadata

Basic usage:

python eest_stateful_generator.py \
    --chain mainnet \
    --fork Prague \
    --rpc-seed-key YOUR_PRIVATE_KEY \
    --rpc-address YOUR_ADDRESS \
    --test-path tests

The tool writes mitm_config.json automatically (edit it if you need to tweak defaults). A generated example looks like:

{
  "rpc_direct": "http://127.0.0.1:8545",
  "engine_url": "http://127.0.0.1:8551",
  "jwt_hex_path": "engine-jwt/jwt.hex",
  "finalized_block": "0x..."
}

Key notes:

• The generator automatically handles node setup, health checks, JWT authentication, and cleanup.
• It uses OverlayFS for ephemeral node state (no mutation of base snapshot, the snapshot can be downloaded once and reused)
• Requires custom Nethermind image: nethermindeth/nethermind:gp-hacked
• Produces deterministic engine payloads without triggering reorgs

Additional options:

• --chain: Target network (mainnet, sepolia, holesky, goerli or ethereum (perfnet))
• --no-snapshot: Skip snapshot download
• --keep: Preserve containers and logs after completion
• --refresh-snapshot: Force re-download of snapshot

Important: This feature is still in development (See PR description). The script writes mitm_config.json automatically; edit it only if you need to customize the generated values.

Contributing: see CONTRIBUTING.md