Skip to content

Commit 4e0d20c

Browse files
KamiDKamiD
committed
Merge branch 'wesley/claude-skills' into 'main'
feat: benchmark and devnet skills See merge request github/xlayer-toolkit!3
2 parents f7ad546 + ab8b74b commit 4e0d20c

4 files changed

Lines changed: 383 additions & 0 deletions

File tree

‎.claude/skills/adventure/SKILL.md‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
---
2+
name: adventure
3+
description: Run the X Layer "adventure" benchmark tool (ERC20 / native / io / fib / create stress tests on an Optimism L2 using 20k accounts). Use when the user asks to run adventure, benchmark X Layer, or run erc20-bench / native-bench / io-bench / fib-bench / create-bench.
4+
---
5+
6+
# Adventure — X Layer Benchmark Tool
7+
8+
Adventure is a stress-testing tool for X Layer (Optimism L2). It uses ~20,000 accounts to drive concurrent ERC20 / native / io / fib / create workloads against a node and measure performance and stability.
9+
10+
## 0. Adventure directory
11+
12+
The adventure project directory defaults to `tools/adventure` under the current repository root (i.e. `<repo-root>/tools/adventure`). No configuration file is needed — resolve the repo root with `git rev-parse --show-toplevel` and use `$REPO_ROOT/tools/adventure` as the working directory for all commands below.
13+
14+
## 1. Commands
15+
16+
All commands must be executed from the adventure directory (`cd "$(git rev-parse --show-toplevel)/tools/adventure" && ...`).
17+
18+
### ERC20 stress test
19+
20+
```bash
21+
make erc20
22+
# equivalent to:
23+
adventure erc20-init 10000ETH -f ./testdata/config.json
24+
adventure erc20-bench -f ./testdata/config.json --contract 0xContractAddress
25+
```
26+
27+
### Native token stress test
28+
29+
```bash
30+
make native
31+
# equivalent to:
32+
adventure native-init 10000ETH -f ./testdata/config.json
33+
adventure native-bench -f ./testdata/config.json
34+
```
35+
36+
### IO bench
37+
38+
```bash
39+
make io
40+
# equivalent to:
41+
adventure simulator-init 10000ETH -f ./testdata/config.json
42+
adventure io-bench -f ./testdata/config.json
43+
```
44+
45+
### Fib bench
46+
47+
```bash
48+
make fib
49+
# equivalent to:
50+
adventure simulator-init 10000ETH -f ./testdata/config.json
51+
adventure fib-bench -f ./testdata/config.json
52+
```
53+
54+
### Create bench
55+
56+
```bash
57+
make create
58+
# equivalent to:
59+
adventure native-init 10000ETH -f ./testdata/config.json
60+
adventure create-bench -f ./testdata/config.json
61+
```
62+
63+
### Preconditions to remind the user about
64+
65+
- RPC node reachable (default `http://127.0.0.1:8123`).
66+
- `testdata/config.json` exists and is tuned for the run.
67+
- `testdata/accounts/accounts-20000.txt` (20,000 accounts) is present as default, but may change if account number change.
68+
- `senderPrivateKey` in `testdata/config.json` has enough balance to fund init.
69+
70+
## 2. Configuration file (`testdata/config.json`)
71+
72+
Editable parameters:
73+
74+
```json
75+
{
76+
"rpc": ["http://127.0.0.1:8123"],
77+
"accounts": 20000,
78+
"senderPrivateKey": "0x...",
79+
"concurrency": 20,
80+
"mempoolPauseThreshold": 50000,
81+
"targetTPS": 0,
82+
"maxBatchSize": 100,
83+
"gasPriceGwei": 100,
84+
"saveTxHashes": false,
85+
"simulatorParams": {
86+
"simulatorConfig": {
87+
"load_accounts": 12,
88+
"update_accounts": 5,
89+
"create_accounts": 0,
90+
"load_storage": 49,
91+
"update_storage": 9,
92+
"delete_storage": 0,
93+
"create_storage": 2,
94+
"fib": 100
95+
}
96+
}
97+
}
98+
```
99+
100+
### Top-level fields
101+
102+
- `rpc` — RPC endpoint URLs (list; transactions are distributed across them).
103+
- `accounts` — number of benchmark accounts used as both senders and receivers. On first run, `testdata/accounts/accounts-<N>.txt` is auto-generated with this many private keys and reused on subsequent runs.
104+
- `senderPrivateKey` — funds contract deployment and token distribution during `*-init` commands.
105+
- `concurrency` — number of concurrent senders.
106+
- `mempoolPauseThreshold` — pause sending when mempool exceeds this size.
107+
- `targetTPS` — target transactions per second; `0` means no rate limit.
108+
- `maxBatchSize` — maximum transactions per batch (default `100`).
109+
- `gasPriceGwei` — gas price in Gwei.
110+
- `saveTxHashes` — when true, writes tx hashes to `./txhashes.log`.
111+
112+
### `simulatorParams.simulatorConfig` (used by `io-bench` / `fib-bench` / `create-bench`)
113+
114+
Parameters passed to the on-chain simulator contract to shape each transaction's workload:
115+
116+
- `load_accounts` — account loads per tx (reads of existing accounts).
117+
- `update_accounts` — account updates per tx.
118+
- `create_accounts` — new accounts created per tx.
119+
- `load_storage` — storage slot reads per tx.
120+
- `update_storage` — storage slot updates per tx.
121+
- `delete_storage` — storage slot deletions per tx.
122+
- `create_storage` — new storage slots written per tx.
123+
- `fib` — Fibonacci iteration count; controls CPU/compute load per tx (used by `fib-bench`).
124+
125+
Treat `senderPrivateKey` as sensitive: never echo or commit it, mask when showing values from the config, and never send it off-machine.
126+
127+
## 3. Behavior
128+
129+
- If the user's request is ambiguous ("run adventure"), ask which workload (erc20 / native / io / fib / create) and whether they want the full `make <target>` flow or a specific step.
130+
- Before running any `make` target, `cd` into `<repo-root>/tools/adventure`.
131+
- Do not modify `testdata/config.json` unless the user asks for a specific change.
132+
- Stream output so the user can watch benchmark progress.

‎.claude/skills/devnet/SKILL.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
---
2+
name: devnet
3+
description: Bring up the X Layer devnet (OP Stack local test environment) end to end — verify repo paths in example.env, build images via devnet/init.sh, start services with ./0-all.sh, then confirm the sequencer is producing blocks on port 8123. Use when the user asks to start / run / deploy / bring up the devnet, or to restart / verify the local OP Stack environment.
4+
---
5+
6+
# Devnet — X Layer local OP Stack bring-up
7+
8+
This skill brings up the local OP Stack devnet in `<repo-root>/devnet`, confirms repo paths are wired up in `example.env`, builds images, starts services, and verifies the L2 sequencer is producing blocks.
9+
10+
See `devnet/README.md` for full background. The flow here is the "first-time setup then run" path, not `make run`.
11+
12+
## 0. Working directory
13+
14+
All commands run from `<repo-root>/devnet`. Resolve the repo root with `git rev-parse --show-toplevel` and `cd "$(git rev-parse --show-toplevel)/devnet"` before each step (or run once at the start). Never edit `.env` directly, .env prepare by user from `example.env`.
15+
16+
## 1. Check repo paths in `example.env`
17+
18+
Before building, verify that the *local repo directories* required for local image builds are configured. If `SKIP_*_BUILD=true` for a component, its path is not needed; if `SKIP_*_BUILD=false` the corresponding `*_LOCAL_DIRECTORY` must be an absolute path to a cloned repo.
19+
20+
Components and their variables in `example.env`:
21+
22+
| Component | Skip flag | Local dir variable | Upstream repo |
23+
|-----------|-----------|---------------------|----------------|
24+
| OP Stack | `SKIP_OP_STACK_BUILD` | `OP_STACK_LOCAL_DIRECTORY` | https://github.com/okx/optimism |
25+
| OP Contracts | `SKIP_OP_CONTRACTS_BUILD` | (uses `OP_STACK_LOCAL_DIRECTORY`) | same as above |
26+
| OP Geth | `SKIP_OP_GETH_BUILD` | `OP_GETH_LOCAL_DIRECTORY` | https://github.com/okx/op-geth |
27+
| OP Reth | `SKIP_OP_RETH_BUILD` | `OP_RETH_LOCAL_DIRECTORY` | https://github.com/okx/reth |
28+
| OP Succinct | `SKIP_OP_SUCCINCT_BUILD` | `OP_SUCCINCT_LOCAL_DIRECTORY` | (optional, only if `PROOF_ENGINE=op-succinct`) |
29+
| Kailua | `SKIP_KAILUA_BUILD` | `KAILUA_LOCAL_DIRECTORY` | (optional, only if `PROOF_ENGINE=kailua`) |
30+
31+
Recommended check (run from `<repo-root>/devnet`):
32+
33+
```bash
34+
grep -E '^(SKIP_[A-Z_]+_BUILD|[A-Z_]+_LOCAL_DIRECTORY)=' example.env
35+
```
36+
37+
For every `SKIP_X_BUILD=false`, confirm the matching `X_LOCAL_DIRECTORY` is a non-empty absolute path that exists on disk. If any required path is missing or empty, **stop and prompt the user** with a clear list of which variables need to be filled in, e.g.:
38+
39+
> `OP_RETH_LOCAL_DIRECTORY` is empty but `SKIP_OP_RETH_BUILD=false`. Please set it to the absolute path of your local `okx/reth` clone in `example.env`, then re-run this skill.
40+
41+
Do not attempt to guess paths or clone repositories automatically.
42+
43+
Also confirm the sequencer / RPC mode combination matches the user's intent (e.g. `SEQ_TYPE=reth` / `RPC_TYPE=reth` vs `geth`). If unclear, ask.
44+
45+
## 2. Build images — `./init.sh`
46+
47+
Once `example.env` is correct:
48+
49+
```bash
50+
cd "$(git rev-parse --show-toplevel)/devnet"
51+
./clean.sh # sync example.env -> .env and stop any stale containers
52+
./init.sh # build Docker images for all components with SKIP_*_BUILD=false
53+
```
54+
55+
Notes:
56+
- `init.sh` can take a long time (tens of minutes) depending on which components are being built locally. Run in the foreground and stream output so the user can see progress.
57+
- If `init.sh` fails, surface the failing step (usually a Docker build) and stop — do not proceed to start services.
58+
- Re-run `init.sh` only when code changes require rebuilt images.
59+
60+
## 3. Start services — `./0-all.sh`
61+
62+
```bash
63+
cd "$(git rev-parse --show-toplevel)/devnet"
64+
./0-all.sh
65+
```
66+
67+
This chains `1-start-l1.sh` → `2-deploy-op-contracts.sh` → `3-op-init.sh` → `4-op-start-service.sh` (and `5-*` / `6-*` if the corresponding proof engines are enabled). Expect it to take several minutes; watch for errors in contract deployment and op-geth/op-reth init.
68+
69+
## 4. Verify block production on port 8123
70+
71+
After `0-all.sh` exits successfully, confirm the L2 sequencer RPC is up and blocks are advancing. Port `8123` is the `op-geth-seq` / sequencer RPC (see the Service Ports table in `devnet/README.md`).
72+
73+
### 4.1 Port is listening
74+
75+
```bash
76+
# macOS
77+
lsof -iTCP:8123 -sTCP:LISTEN -n -P
78+
# or
79+
nc -z localhost 8123 && echo "8123 open" || echo "8123 closed"
80+
```
81+
82+
If the port is not listening, inspect container state:
83+
84+
```bash
85+
cd "$(git rev-parse --show-toplevel)/devnet"
86+
docker compose ps
87+
docker compose logs --tail=200 op-reth-seq
88+
```
89+
90+
Report the failing container and stop — do not claim success.
91+
92+
### 4.2 Block number is increasing
93+
94+
Query `eth_blockNumber` several times with a short gap and confirm the value strictly increases. Use `curl` (no extra tooling required):
95+
96+
```bash
97+
for i in 1 2 3 4 5; do
98+
curl -s -X POST -H 'Content-Type: application/json' \
99+
--data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
100+
http://localhost:8123 \
101+
| sed -E 's/.*"result":"(0x[0-9a-fA-F]+)".*/\1/'
102+
sleep 2
103+
done
104+
```
105+
106+
Pass criteria:
107+
- Every response is a valid `0x...` hex block number (not an error object).
108+
- The decimal value strictly increases across samples (at least one increment over ~10 seconds).
109+
110+
If the value does not change, the sequencer is not producing blocks. Check `docker compose logs op-geth-seq` and `docker compose logs op-node` (or `op-seq`), report the symptom, and stop. Do not report the devnet as healthy.
111+
112+
### 4.3 Reporting
113+
114+
When all three checks pass (port listening, valid block numbers, strictly increasing), report concisely:
115+
116+
> Devnet is up. Port 8123 listening. Block number advanced from `<n0>` to `<nN>` over `<seconds>`s.
117+
118+
## 5. Things to avoid
119+
120+
- Do not edit `.env` directly — always edit `example.env`, then `./clean.sh`.
121+
- Do not run `./init.sh` repeatedly "just in case" — it is slow and destructive to image caches.
122+
- Do not skip the block-number verification and assume success from port 8123 being open; a stuck sequencer will still hold the port.
123+
- Do not invent repo paths or auto-clone missing repositories — prompt the user instead.
124+
- Do not print or commit any private keys or funded test-account secrets from `devnet/` configs.

‎.claude/skills/sa-benchmark/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
config.env

‎.claude/skills/sa-benchmark/SKILL.md‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
---
2+
name: sa-benchmark
3+
description: Run the SA-Benchmark (Smart Account / OKX Pay ERC-4337) stress test using the "Old Method" — `./1-setup.sh` (TypeScript deploy + clone/build polycli) followed by `./2-bench.sh` (polycli loadtest erc4337). Use when the user asks to run SA-Benchmark, benchmark OKX Pay / Smart Account / ERC-4337, or run polycli erc4337 loadtest. The SA-Benchmark project lives outside this repo; its absolute path is stored in a per-skill config file and can be changed at any time.
4+
---
5+
6+
# SA-Benchmark — Smart Account / OKX Pay ERC-4337 loadtest
7+
8+
This skill runs the **Old Method** from the SA-Benchmark project README: TypeScript contract deploy + initial UserOperation, followed by a Go `polycli loadtest erc4337` run. The SA-Benchmark repo is external to this toolkit, so its absolute path must be configured before the skill can run.
9+
10+
See `<SA_BENCHMARK_DIR>/README.md` in the target repo for full background. The relevant scripts are `1-setup.sh` and `2-bench.sh` in that directory's root.
11+
12+
## 0. Locate / configure the SA-Benchmark directory
13+
14+
The SA-Benchmark directory is **not** inside this repo. The skill stores its absolute path in:
15+
16+
```
17+
<repo-root>/.claude/skills/sa-benchmark/config.env
18+
```
19+
20+
Contents (single line):
21+
22+
```
23+
SA_BENCHMARK_DIR=/absolute/path/to/SA-Benchmark
24+
```
25+
26+
### First-time initialization
27+
28+
1. Resolve the config path: `CONFIG_FILE="$(git rev-parse --show-toplevel)/.claude/skills/sa-benchmark/config.env"`.
29+
2. If `CONFIG_FILE` does **not** exist, or exists but `SA_BENCHMARK_DIR` is empty / missing:
30+
- **Stop and prompt the user**. Ask for the absolute path to their local SA-Benchmark clone (e.g. `/Users/oker/go/bin/code/ethereum/SA-Benchmark`). Do not guess, do not `find`, do not auto-clone.
31+
- Once the user replies, validate that the path is absolute, exists, is a directory, and contains both `1-setup.sh` and `2-bench.sh` and an `example.env` file. If any check fails, report the exact problem and re-prompt.
32+
- Write the config file with `SA_BENCHMARK_DIR=<path>` and confirm.
33+
3. If `CONFIG_FILE` exists and `SA_BENCHMARK_DIR` points at a valid directory, proceed.
34+
35+
### Changing the directory later
36+
37+
The user can change the path at any time by saying things like "change SA-Benchmark dir to X", "point SA-Benchmark at /new/path", or "reconfigure SA-Benchmark". When that happens:
38+
1. Validate the new path (same checks as first-time init).
39+
2. Overwrite `config.env` with the new `SA_BENCHMARK_DIR=...`.
40+
3. Confirm the updated value back to the user. Do **not** delete or clean up the previous directory.
41+
42+
Never edit the SA-Benchmark project's `.env` or scripts to change its location — the path is a Claude-side config only.
43+
44+
## 1. Prepare `.env` in the SA-Benchmark directory
45+
46+
All remaining steps run with `cd "$SA_BENCHMARK_DIR"`.
47+
48+
1. If `.env` does not exist, copy it from the template: `cp example.env .env`. Do not overwrite an existing `.env`.
49+
2. Verify the required fields for the Old Method are populated in `.env`:
50+
- `PRIVATE_KEY` — funded L2 account (must not be empty / default)
51+
- `LOCAL_RPC_URL` — L2 RPC, typically `http://127.0.0.1:8123` when running against the local devnet
52+
- `CARD_ENABLE` — must be `false` (or unset). The Old Method is the ERC4337 + polycli path; `CARD_ENABLE=true` switches `1-setup.sh` / `2-bench.sh` to the Card Claim path, which is **not** what this skill runs.
53+
- `POLYCLI_REPO`, `POLYCLI_BRANCH` — used by `1-setup.sh` to clone and build polycli
54+
- `GAS_PRICE`, `CALLDATA_FILE`, `CALLDATA_SIZE`, `TOTAL_UOP`, `BATCH_SIZE`, `CONCURRENCY`, `RATE_LIMIT`, `CALLDATA_TYPE`, `VERIFIER_TYPE` — required by `1-setup.sh`'s pre-flight check; empty values will make the script exit.
55+
3. If any required field is missing, **stop and list them** for the user. Only print each variable's *name*, never its value — secrets like `PRIVATE_KEY` must not be echoed.
56+
57+
Quick check (safe — prints only variable *names* whose values are empty):
58+
59+
```bash
60+
cd "$SA_BENCHMARK_DIR"
61+
[ -f .env ] || cp example.env .env
62+
set -a; . ./.env; set +a
63+
for v in PRIVATE_KEY LOCAL_RPC_URL POLYCLI_REPO POLYCLI_BRANCH GAS_PRICE \
64+
CALLDATA_FILE CALLDATA_SIZE TOTAL_UOP BATCH_SIZE CONCURRENCY \
65+
RATE_LIMIT CALLDATA_TYPE VERIFIER_TYPE; do
66+
[ -z "${!v}" ] && echo "MISSING: $v"
67+
done
68+
[ "${CARD_ENABLE:-false}" = "true" ] && echo "WARNING: CARD_ENABLE=true switches to the Card path, not the Old Method"
69+
```
70+
71+
If the user wants the benchmark to target this toolkit's local devnet, `LOCAL_RPC_URL` should be `http://127.0.0.1:8123` and the devnet must already be up (see the `devnet` skill). Do not start the devnet from inside this skill — if the user wants both, suggest running the `devnet` skill first.
72+
73+
## 2. Setup — `./1-setup.sh`
74+
75+
```bash
76+
cd "$SA_BENCHMARK_DIR"
77+
./1-setup.sh
78+
```
79+
80+
What it does (Old Method branch, i.e. `CARD_ENABLE=false`):
81+
- Clones or updates `POLYCLI_REPO` at `POLYCLI_BRANCH`, then runs `make install` so the `polycli` binary lands in `~/go/bin`.
82+
- `yarn` to install Node dependencies.
83+
- `yarn run deploy` to deploy / attach to the Smart Account + OKX Pay contracts.
84+
- `yarn run senduop:local` to send the initial UserOperation that counterfactually deploys sender accounts and funds bundlers.
85+
86+
Notes:
87+
- The setup can take several minutes (Go build + Node install + on-chain deploys). Stream output; don't background it.
88+
- If `1-setup.sh` fails, surface the failing step and **stop** — do not proceed to `2-bench.sh`. Re-running setup after the user fixes the underlying issue is safe.
89+
- The script populates contract addresses (`ENTRYPOINT`, `ACCOUNT_FACTORY`, `PAY`, `TEST_ERC20`, `WEBAUTHN_VALIDATOR`, …) in `.env`; `2-bench.sh` reads them. If any of those are empty after setup, setup did not finish — investigate before benching.
90+
91+
## 3. Benchmark — `./2-bench.sh`
92+
93+
```bash
94+
cd "$SA_BENCHMARK_DIR"
95+
./2-bench.sh
96+
```
97+
98+
Behavior (Old Method branch):
99+
- Prints the test parameters (TOTAL_UOP, BATCH_SIZE, CONCURRENCY, RATE_LIMIT, CALLDATA_TYPE).
100+
- Writes raw output to `result_<YYYYMMDD_HHMM>.out` in the SA-Benchmark directory.
101+
- Runs `polycli loadtest erc4337` with the contract addresses set up in step 2.
102+
- At the end, strips ANSI escapes, greps the last `tps=...` line, and prints `Final TPS: <value>`.
103+
104+
Heads-up to the user (optional, only mention if the benchmark is expected to run long):
105+
- Perf profile capture: `curl http://localhost:6060/debug/pprof/profile?seconds=120 > prof_<timestamp>.bin`
106+
- Sequencer log monitor: `docker logs op-reth-seq --tail 10 -f 2>&1 | grep TotalDuration-batch`
107+
108+
## 4. Reporting
109+
110+
After `2-bench.sh` exits:
111+
1. Identify the newest `result_*.out` in `$SA_BENCHMARK_DIR` (the one the script just wrote).
112+
2. Extract the final TPS. Prefer the line `TPS: <value>` appended at the end of the file; fall back to the last `tps=<value>` match.
113+
3. Report concisely to the user, e.g.:
114+
> SA-Benchmark finished. Final TPS: `<value>`. Full log: `<SA_BENCHMARK_DIR>/result_<timestamp>.out`.
115+
116+
If `Final TPS` is missing or empty, the loadtest did not complete cleanly — do not claim success. Summarize the tail of the result file (last ~20 lines) and any error lines you can see.
117+
118+
## 5. Things to avoid
119+
120+
- **Never** print, log, or commit `PRIVATE_KEY`, derived wallets, bundler private keys, or any value from `.env`. Reference fields by name only.
121+
- Never clone the SA-Benchmark repo automatically. If the configured path is missing, prompt the user.
122+
- Never run `./loadtest.sh` or `yarn deploy` directly from this skill — that is the "New Method" and is explicitly out of scope here.
123+
- Never set `CARD_ENABLE=true` or run the card path; that is a different benchmark and not what this skill targets.
124+
- Never run `2-bench.sh` before `1-setup.sh` succeeds; the contract addresses it reads from `.env` won't be populated.
125+
- Do not delete or truncate previous `result_*.out` / `result_*.out.bak` files unless the user asks.
126+
- Do not edit the SA-Benchmark project's files to hard-code paths or secrets — the directory path lives only in this skill's `config.env`.

0 commit comments

Comments
 (0)