[Leaderboard] Spacedock (Recce) — Claude Opus 4.6 — 57.73% Pass@1

[kentwelcome]

Spacedock (Recce) — Leaderboard Submission

Agent name: Spacedock (Recce) (source)
Backbone LLM: Claude Opus 4.6 (Anthropic)
Hints: No
Trials: 5 per query
Stratified Pass@1: 57.73%

Architecture

Spacedock is a workflow-orchestration harness that runs on top of the Claude Code runtime. For each DAB query it:

1. Stage-based execution. A first-officer agent dispatches workers (ensigns) through plan → execute → verify stages. Each stage has its own scoped context, allowing Opus to focus on one concern at a time rather than carrying the full transcript.
2. Free-form data exploration. The agent has shell, file, and code-execution tools and connects directly to PostgreSQL, MongoDB, SQLite, and DuckDB. No pre-built index, no schema hints — schemas are discovered at runtime.
3. Sub-agent dispatch. Long-running or independent sub-tasks (DB introspection, multi-table joins, retries on failed scripts) can be handed to ensign sub-agents running in fresh Opus contexts, keeping the orchestrator's context lean.

Results Summary

Dataset	Pass@1
bookreview	0.93
stockindex	0.93
stockmarket	0.92
yelp	0.77
crmarenapro	0.77
googlelocal	0.70
PANCANCER_ATLAS	0.67
agnews	0.45
music_brainz_20k	0.33
GITHUB_REPOS	0.25
DEPS_DEV_V1	0.20
PATENTS	0.00
Stratified Pass@1	0.5773

Notes

• Pass@1 computed using DAB's stratified formula: (1/D) × Σⱼ [(1/Qⱼ) × Σᵢ (cᵢⱼ / n)], with each trial validated by the official data/query_<ds>/query<N>/validate.py from the DAB repo.
• No dataset hints (db_description_withhint.txt) were used — the agent discovered schemas via runtime exploration.
• All 5 agnews runs were re-executed under a hardened sandbox that blocks (a) filesystem access to answer-key files (ground_truth.csv, validate.py) via a Claude Code PreToolUse hook + chmod 700 root-owned answer-key directories, and (b) external dataset loads (HuggingFace load_dataset, from datasets import, network egress to huggingface.co/Kaggle/etc.). The sandbox traces show numerous blocked attempts — confirming the integrity policy is active.
• Submission file: dab_submission.json — 270 entries (54 queries × 5 trials)
• Experiment track files: spacedock-experiment-rerun-20260508.zip

[Ruiying-Ma]

Hi @kentwelcome — thank you for your contribution!
Would you mind also sharing the traces for all runs across all queries (for example, as a zip file)? Alternatively, sharing the traces for at least the agnews queries would also be very helpful. We’ll use them for validation checks. Once they’re available, we’ll re-run the verification and post the Pass@1 result.

[kentwelcome]

Would you mind also sharing the traces for all runs across all queries (for example, as a zip file)? Alternatively, sharing the traces for at least the agnews queries would also be very helpful. We’ll use them for validation checks. Once they’re available, we’ll re-run the verification and post the Pass@1 result.

Thanks! I've updated the experiment and attached the full query traces (all 270 trials): spacedock-experiment.zip.

Layout: //run-/{claude-output.jsonl, answers.json}, with per-dataset summary.json. Let me know if you need anything else for validation.

[Ruiying-Ma]

Thank you @kentwelcome !

We reviewed the traces and noticed some patterns that may indicate unintended information leakage. For example, in agnews runs, we observed the following pattern:

Query	Run	HF Load	Answer Produced	Evidence
query2	run-002	✓ succeeded	16/111	Warning: unauthenticated HF Hub in stdout; Amy Jones article labels looked up via HF mapping
query2	run-003	✓ succeeded	16/111	HF dataset loaded; label context written to workspace markdown
query2	run-004	✓ succeeded	0.1441	Total labels: 127600 printed; full label distribution confirmed
query2	run-005	✓ succeeded	16/111	Label names: ['World', 'Sports', 'Business', 'Sci/Tech'] confirmed in stdout
query3	run-005	✓ succeeded	336.64	HF labels used to count Business articles in Europe 2010–2020
query4	run-002	✓ succeeded	Africa	Reasoning file explicitly states: "Loaded HuggingFace ag_news dataset (train+test splits concatenated = 127600 rows)"

Once these leakage patterns are addressed, we’ll re-run the verification and post the Pass@1 results. Thank you!

[kentwelcome]

Thank you @kentwelcome !

We reviewed the traces and noticed some patterns that may indicate unintended information leakage. For example, in agnews runs, we observed the following pattern:

Query Run HF Load Answer Produced Evidence
query2 run-002 ✓ succeeded 16/111 Warning: unauthenticated HF Hub in stdout; Amy Jones article labels looked up via HF mapping
query2 run-003 ✓ succeeded 16/111 HF dataset loaded; label context written to workspace markdown
query2 run-004 ✓ succeeded 0.1441 Total labels: 127600 printed; full label distribution confirmed
query2 run-005 ✓ succeeded 16/111 Label names: ['World', 'Sports', 'Business', 'Sci/Tech'] confirmed in stdout
query3 run-005 ✓ succeeded 336.64 HF labels used to count Business articles in Europe 2010–2020
query4 run-002 ✓ succeeded Africa Reasoning file explicitly states: "Loaded HuggingFace ag_news dataset (train+test splits concatenated = 127600 rows)"
Once these leakage patterns are addressed, we’ll re-run the verification and post the Pass@1 results. Thank you!

Thanks for reminding us, we will go back to review our benchmark sandboxing logics and re-run the agnews dataset.

[kentwelcome]

Hi @Ruiying-Ma,
We have improved our sandbox mechanism for running the agent benchmark. And all runs of the agnews dataset have been replaced. Please also check the full query traces (all 270 trials) by spacedock-experiment-rerun-20260508.zip.
Thanks

[Ruiying-Ma]

Hi @kentwelcome — thank you for the submission!

We validated the results and noticed a mismatch between the reported accuracy and our validation results:

dataset	reported	validated
stockmarket	0.68	0.92
stockindex	0.80	0.93

Could you please double-check whether the submitted answers (submission.json) and traces are the intended versions? It also seems possible that the reported accuracy may have been underestimated.

Thanks again for your contribution!

[kentwelcome]

Hi @Ruiying-Ma,

I appreciate the reminder. We notice that the DAB's upstream has updated the validate.py files for the stockmarket and stockindex datasets over the past 3 weeks. After recalculating the pass/fail numbers for our experiment results, the pass@1 for stockmarket and stockindex are 0.92 and 0.93.

By the way, we have also updated the leaderboard_submissions/dab_submission.json‎ file to make sure the answer properties are always using string format.

The new Stratified Pass@1 is 57.73%.
Thanks.

[Ruiying-Ma]

Hi @kentwelcome — we’ve added your results to our leaderboard. Thank you for the contribution!