BBox Tester

Benchmark screenshot-based GUI grounding across multimodal model providers.

bbox-tester sends an image and target instruction to a provider/model matrix, extracts [y1, x1, y2, x2] boxes in normalized 0-1000 coordinates, scores the prediction against a gold box, and writes JSON/CSV/HTML reports with overlays.

Install

uv sync

Optional provider credentials go in .env or your shell environment. Start from:

cp .env.example .env

Quick Start

Validate the sample suite and expanded default model matrix:

uv run bbox-tester \
  --suite benchmarks/sample-yahoo/tasks.jsonl \
  --providers ollama \
  --dry-run

Equivalent smoke-test wrapper:

scripts/smoke-dry-run.sh

Run all smoke wrappers without making provider calls:

scripts/smoke-all.sh

Run all smoke wrappers and execute the Ollama benchmark:

BBOX_TESTER_RUN_OLLAMA=1 scripts/smoke-all.sh

Run the sample suite:

uv run bbox-tester \
  --suite benchmarks/sample-yahoo/tasks.jsonl \
  --providers ollama \
  --model gemma4:latest \
  --runs 3

Equivalent Ollama smoke wrapper:

scripts/smoke-ollama.sh

Run the sample Yahoo benchmark against the full default provider/model matrix:

scripts/sample-all-providers.sh

This defaults to BBOX_TESTER_CONCURRENCY=4 and 60 second provider timeouts so one slow model does not make the run look frozen.

Preview the full expanded matrix without calling providers:

scripts/sample-all-providers.sh --dry-run

Single-image mode requires an explicit gold box:

uv run bbox-tester \
  --image benchmarks/sample-yahoo/yahoo.png \
  --target "Output the position of yahoo!finance" \
  --gold-box 17 203 34 259 \
  --providers gemini

Providers

Supported providers:

• azure: Azure OpenAI Responses API deployments.
• gemini: Gemini Developer API or Vertex AI.
• openrouter: OpenRouter through LiteLLM.
• ollama: local Ollama vision models.

The default model matrix lives in bbox_tester/models.py. Use --config config.json only when you need to override model names or reasoning settings.

Example config:

{
  "models": [
    {
      "provider": "ollama",
      "model": "gemma4:latest",
      "reasoning": ["auto"]
    }
  ]
}

Task Suites

Suites are JSONL or JSON files. Relative image paths resolve from the suite file location.

{"id":"yahoo_finance_logo","image":"yahoo.png","target":"Output the position of yahoo!finance","gold_box":[17,203,34,259]}

Required fields:

• id: stable task identifier.
• image: screenshot path.
• target or instruction: object or region to locate.
• gold_box: expected [y1, x1, y2, x2] box in normalized 0-1000 coordinates.

Optional fields:

• expected_click: [x, y] point in normalized 0-1000 coordinates.
• source, license, notes: publication metadata.

Outputs

Each run writes a timestamped folder under runs/ unless --output-dir is set.

• summary.json: structured run payload.
• results.jsonl: one provider call per line.
• results.csv: spreadsheet-friendly per-run metrics.
• leaderboard.csv: aggregate metrics by provider/model/reasoning.
• leaderboard.md: GitHub-ready aggregate table.
• report.html: visual report.
• annotated/*.jpg: screenshot overlays.

Raw provider responses and prompts are omitted by default. Add --include-raw-responses only for local debugging on non-sensitive data.

Coordinate Checker

Use bbox-check to score one predicted box or click without calling providers:

uv run bbox-check --gold-box 17 203 34 259 --pred-box 19 205 33 257
uv run bbox-check --gold-box 17 203 34 259 --pred-click 230 25

Equivalent smoke checker:

scripts/smoke-check.sh

For click mode, the verdict is based on whether the point lands inside the gold box.

Development

uv run ruff format .
uv run ruff check .
uv run pytest

Spec-driven project docs live in spec/. Agent-facing repository context lives in .gemini/agent.md.

Build a wheel:

uv build --wheel

Generated outputs, virtualenvs, caches, .env, and service-account files are ignored and should stay out of the project tree when possible.

Publication Note

The included Yahoo screenshot is a smoke-test sample marked review-before-publication. Replace it with licensed or synthetic data before using this repository as a public benchmark.