Repository navigation
feat: adapter — run AssistantBench (realistic time-consuming live-web tasks) under the ClawBench harness #188
Description
Goal
Let the ClawBench harness load + execute tasks from oriyor/assistantbench — "Can Web Agents Solve Realistic and Time-Consuming Tasks?" Live-web tasks intentionally chosen to be long-horizon (e.g. multi-step research, comparison shopping); HF dataset at AssistantBench/AssistantBench.
Scope
- Task loader —
clawbench.corpus.adapters.assistantbenchingests their HF dataset; normalizes their(task, answer_type, gold_answer, reference_links)schema into our task format. - Answer-grading — their scoring is answer-string-based (with answer-type rubric for numeric/date/list), not HTTP-interception. Surface their grader as a third stage in
run-meta.json(after our Stage 1 + Stage 2) so adopters who care about answer-correctness alongside endpoint-correctness see both. - Time-limit handling — their tasks are designed long-horizon (many minutes). Honor each task's recommended
max_stepsinstead of our default 30-min wallclock. - CLI —
clawbench run --corpus assistantbench --model <m>end-to-end.
Why now
AssistantBench targets long-horizon live web research — a slice ClawBench V1+V2 doesn't directly cover. Hosting it under our harness extends the difficulty curve (most V2 tasks are ≤5 min; AssistantBench tasks routinely exceed 15 min) and gives us a third independent metric (answer-correctness vs interception vs LLM payload judge).
Acceptance
-
clawbench run --corpus assistantbench --limit 5 --model <m>produces 5-layer bundles -
run-meta.jsonincludesassistantbench_answer_scorealongside ourintercepted/judge_match - Match upstream
eval/eval_answer.pyoutputs within ±2pp on a sampled subset - Docs:
eval/adapters/assistantbench.md
Activity
Merge pull request #259 from TIGER-AI-Lab/feat/answer-judge
nahatav commentedon Sep 23, 2026
Opened #379 against this.
It covers the first two bullets of the scope: the task loader (
clawbench-assistantbench-adapt, normalizing(id, task, answer, gold_url, explanation, difficulty)into our task format) and the answer grader surfaced as a third stage (clawbench-assistantbench-score, writingassistantbench_answer_scoreintorun-meta.jsonnext tointerceptedandjudge_match). Long-horizon time limits are a--time-limitflag defaulting to 30 minutes.The design decision worth your review: AssistantBench has no interceptable final request, and rather than add the
eval_schema.modediscriminatordocs/answer-mode-tasks.mdsketches, the adapter reuses the answer-submit path the bundled claw-eval suite already uses on main, verbatim. That means no runner changes at all, and a test pins the contract against a bundled claw-eval task so the two ports cannot drift.On the third acceptance bullet: I could not do a live
±2pprun, so instead I checked the metric itself against the upstream evaluator under numpy 2.5.3 and scipy 1.18.1 over 7249 (prediction, gold) pairs. Exact agreement to 1e-9 on all 6581 pairs upstream can score; the other 668 are malformed predictions where upstream raises and we score 0 rather than taking down the batch. That run caught two real bugs in my first draft, both now fixed and pinned by tests. Details and the numbers are in the PR.Not attempted, and said so in
docs/assistantbench.md: the held-out test split ships no answers so the deterministic metric cannot score it locally, and--corpus assistantbenchas a first-class selector is left to #72 since this ships as a suite generator plus--cases-dir, matching how harbor and edgebench already work.