vl-benchmark-tool

Setup

npm install
cp .env.example server/.env  # fill in API keys

Running

From the repo root, one command runs both workspaces via concurrently (prefixed server/client output, Ctrl-C stops both):

npm run dev

Or run them in separate terminals — useful when you want to restart or read the logs of just one:

# Terminal 1 — server
cd server && npm run dev

# Terminal 2 — client
cd client && npm run dev

MongoDB is expected at localhost:27017 (see docker-compose.yml, or npm run docker:up).

Ollama / Code Server

Requires the ollama Docker Compose profile.

One-time: build the sandbox image before starting the stack:

docker build -f code_server/Dockerfile.sandbox -t python-sandbox code_server/

Then start the stack:

docker compose --profile ollama up -d

Rebuild code_server only (e.g. after code changes):

docker compose --profile ollama up -d --build code_server

The python-sandbox image is not managed by Compose — it must be rebuilt manually whenever code_server/Dockerfile.sandbox changes.

Dataset files are sent to the sandbox as base64 payloads at execution time — no volume mounts required.

Maintenance Scripts

server/scripts/swap-user.ts

Users are keyed by name only, so deleting and recreating a user (or creating a duplicate under a different spelling) yields a new _id and orphans everything the old _id owned — plans, executions, verifier configs, benchmarks, votes. This script rewrites those references onto the surviving user instead of losing the data.

npx tsx server/scripts/swap-user.ts <fromId> <toId>

It aborts before writing if both users have a human vote on the same run result (a unique index forbids that); delete one of the two votes and re-run. It does not delete the stale user document.

Benchmark Buckets

A bucket is the unit of visual content within a benchmark. It groups one or more plots with a set of questions that refer to those plots.

Field	Description
images[]	Paths to uploaded image files (one per chart/plot). Empty = image input mode unavailable.
datasets[]	Paths to uploaded CSV/JSON files (one per chart). Empty = dataset input mode unavailable.
chartPrompt	Required when datasets is non-empty. Describes the chart(s) — referenced by the executor as the system prompt in dataset modes. You can refer to individual datasets by their filename (e.g. sales.csv).
chartType	Optional label (e.g. "bar chart", "dashboard"). Informational only.
items[]	Questions about the plot(s). Each item has a question, optional answerOptions, groundTruth, taskType.

Single-chart vs. dashboard: there is no explicit mode flag. A bucket with one image/dataset is a standard single-chart benchmark item; a bucket with multiple images/datasets is a dashboard. The executor attaches all images in image mode, and concatenates (or uploads) all datasets in dataset modes.