Chat3D Benchmark

Chat3D Benchmark evaluates how conversational AI agents reconstruct structured, game-ready 3D assets from standardized multi-view reference boards. It compares geometry, semantic structure, editability, reliability, and generation efficiency against hidden PartNet-Symh ground truth.

The benchmark is command-line based and does not require Jupyter or Google Colab.

Project status: Early research release. The current code supports pilot-scale experiments; benchmark results should report uncertainty and should not be treated as a universal ranking of 3D-generation systems.

The current generator adapters cover:

• img2threejs
• mini-Articraft
• Roblox Procedural Models

Repository layout

src/agentic_3d_benchmark/  Python package and CLI
adapters/                  Native-output adapters for the three systems
configs/                   Example manifests and pilot selections
docs/                      Experimental protocol and project documentation
tests/                     Fast unit and smoke tests
benchmark_data/            Generated locally; ignored by Git
benchmark_outputs/         Generator submissions; ignored by Git

The PartNet-Symh dataset stays outside this repository. This keeps hidden ground truth and large meshes out of version control. Generated reference boards and submissions are also ignored by Git by default.

Setup

Python 3.10 or newer is required.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[dev]'

Quick start: Bag pilot

Download PartNet-Symh separately and identify the directory for the category you want to evaluate. The examples below use a shell variable so they work on any machine:

export CHAT3D_DATA_ROOT=/path/to/PartNet-Symh/Bag

Validate the downloaded category:

agentic-3d inspect \
  --data-root "$CHAT3D_DATA_ROOT" \
  --category Bag

Select five diverse objects and render front/side/top reference boards:

agentic-3d prepare \
  --data-root "$CHAT3D_DATA_ROOT" \
  --category Bag \
  --pilot-size 5 \
  --output benchmark_data

This creates inventory.csv, pilot_selection.csv, task_map.json, the reference boards, and separate method task sheets:

benchmark_data/method_tasks/
├── img2threejs/<task_id>.md
├── articraft/<task_id>.md
├── roblox_procedural_models/<task_id>.md
└── index.json

Every method directory contains a human-readable Markdown sheet and a machine-readable JSON record for each object. Give only the generated task sheet and reference board to the agent. Never expose the PartNet OBJ, annotations, or other ground-truth-derived files during generation.

Package generator results

Follow ADAPTERS.md to turn each system's native output into the common bundle:

benchmark_outputs/<system>/<task_id>_run<run>/
├── model.glb or model.obj
├── structure.json
├── generation.json
└── submission.json

Collect those bundles into one manifest:

python adapters/collect_submissions.py \
  --outputs benchmark_outputs \
  --task-map benchmark_data/task_map.json \
  --manifest benchmark_data/manifest.json

Evaluate all recorded attempts, including failures:

agentic-3d evaluate \
  --manifest benchmark_data/manifest.json \
  --output benchmark_data/results.csv

The command also writes results_summary.csv. Use --samples to change the surface-sampling budget from its default of 10,000 points.

Reproducibility records

For every run, record the reconstruction method, operating agent, exact model version when available, prompt, reference-board hash, seed, elapsed time, number of attempts, cost, and manual intervention. Keep method and agent identities separate—for example, img2threejs is a method while Codex and Claude are agents.

Test

pytest
ruff check .

Experimental protocol

The complete protocol is documented in docs/experiment-protocol.md. In brief: freeze inputs and metrics before generation, keep hidden ground truth isolated, retain failed attempts, and record enough provenance to reproduce every run.

Contributing

See CONTRIBUTING.md for development setup, validation, and rules for changing benchmark behavior without invalidating ongoing experiments.

License

The benchmark code is provided under the MIT License. PartNet-Symh, generator outputs, and third-party tools remain subject to their respective licenses. Cite the dataset authors when publishing results.

Dataset attribution

This project uses the external PartNet-Symh dataset for local ground truth. Dataset files are not distributed with this repository. Consult the upstream repository for its license, download instructions, and required academic citations.