Repository navigation
eval(headless): benchmark Maka, Codex, and Claude Code synchronously on DeepSeek V4 Flash #1970
Description
Problem
Users want a head-to-head comparison of Maka, Codex CLI, and Claude Code on the same DeepSeek V4 Flash runtime. The current headless harness supports only two arms per experiment, has no pinned Claude Code competitor adapter, and supports Codex only in its existing OpenAI composition. Two independent A/B runs would duplicate the Maka baseline and expose the result to round-to-round provider and infrastructure drift.
DeepSeek documents official integrations for both Claude Code and Codex, but the benchmark path does not yet implement those integration contracts.
Desired outcome
Extend the existing packages/headless Harbor/Pier path so one frozen Terminal-Bench 2.1 experiment schedules Maka, Codex CLI, and Claude Code as three synchronized arms for every task. All arms must use deepseek-v4-flash with the same task instruction, task snapshot, verifier, effort level, deadline policy, and declared execution policy; each agent retains its native system prompt and tool semantics.
The infrastructure must:
- support native three-arm scheduling, resume identity, manifests, and reports without creating a parallel benchmark harness;
- pin and verify an offline Linux/x64 Claude Code toolchain, route its Anthropic-compatible traffic through the existing credential proxy, and normalize its evidence into the existing cell contract;
- configure the pinned Codex CLI with DeepSeek's official Responses API model catalog while keeping the real API key out of config files, command lines, logs, and task containers;
- record comparable usage, cost, duration, tool-call, failure, and execution-identity evidence for all three arms.
Completion is accepted when a predeclared canary passes for all three arms and one full synchronized Terminal-Bench 2.1 run produces a checked-in docs/eval/ report containing per-arm Pass@1, token/cache usage, cost, duration, infrastructure failures, and per-task outcomes. Its manifest must record the benchmark snapshot, model/provider, effort, agent versions, toolchain and model-catalog fingerprints, and execution policy. Automated tests must cover both competitor integrations, secret boundaries, three-arm scheduling/resume, and report generation.
Alternatives or workarounds
Run Maka vs Codex and Maka vs Claude Code as separate A/B experiments. This reuses the current two-arm path, but it is not a synchronized three-way comparison and makes the shared Maka baseline vulnerable to run-to-run drift.
Activity
Astro-Han commentedon Aug 3, 2026
Claiming this. I’ll implement the synchronized three-arm harness on the existing
packages/headlesspath: explicit agent/provider wire routing, DeepSeek support for Codex and a pinned Claude Code adapter, cohort scheduling with the existing pairwise summaries as projections, and the combined report/validation. I’ll open a draft PR when the implementation and canary-ready checks are complete.fix(headless): classify a trial by who ended it, not by how the excep…
test(headless): pin arm-symmetric deadline reporting at the executabl…
fix(headless): classify a benchmark trial by who ended it, not by how…
fix(headless): resolve host.docker.internal for Harbor's in-container…
fix(headless): stop scope teardown from rewriting the agent's failure
fix(headless): keep a cleanly ended cell whose last stream was torn down
fix(headless): stop misfiling graded competitor cells as infrastructu…
fix(headless): make Maka's settlement window reachable at the Harbor …
fix(headless): honour the command-timeout floor over the Harbor bridge (
fix(headless): bound teardown and mark the output replay (#2151)
Astro-Han commentedon Aug 4, 2026
Run log: infrastructure defects and operational interventions
Recording what happened while producing the Terminal-Bench 2.1 numbers, so the report does not have to carry it. Everything here is either a harness defect we found and fixed, or an operational action taken outside git that a reader would need to know before reproducing the run.
Harness defects found during the run
All three were in the shared command-scope teardown, and all three were arm-directional: they can only fire for arms whose agent runs inside the task container, so the in-container arms absorbed failures the host-side arm could not.
Teardown rewrote the agent's own exception (fix(headless): stop misfiling graded competitor cells as infrastructure failures #2090).
cleanup_process_scoperaised from afinallywhile the agent's exception was in flight, so anAgentTimeoutErrorreached Harbor asNonZeroAgentExitCodeError— a graded timeout recorded as an infrastructure failure. 17 claude-code cells in v7 were classified this way.The output replay had no teardown handle (fix(headless): reap the scoped command's output replay #2146). The wrapper captures a command's output and replays it after the command exits. When an agent-phase timeout stops the caller reading, that replay blocks in
pipe_writeforever and the exec never returns.Teardown itself was unbounded (fix(headless): bound teardown and mark the output replay #2151). Harbor's
_run_docker_compose_commandwithtimeout_sec=Noneis a bareawait process.communicate(). A teardown that hung took the whole trial with it. Confirmed in production: two stalled cells'trial.logends atRunning command: <TERM cleanup>with no completion line, and their containers held zero scope leftovers — the hang was host-side, in the teardown exec. Observed stall: 1524 s.fix(headless): bound teardown and mark the output replay #2151 also fixed two defects found in review: the bridged executor in
maka_agent.pyhad its own unbounded teardown exec, and the cleanup order let the wrapper fork a replay after the sweep had already passed it.Interventions outside git
docker killthe container → "unstick" (kill scope leftovers, keep the container so the verifier can still run) → observe-only. Log:~/.maka/eval/watchdog.log.632b83c7while a run was in flight; cells that started before and after that point ran different code. A second deploy (93470383, containing fix(headless): bound teardown and mark the output replay #2151) was done only after stopping the run first, so no run spans it./etc/docker/daemon.jsonedited on the VM, twice and out of git: an address-pool change, and"debug": truefor forensics. The debug flag is still enabled and must be reverted; backup at/etc/docker/daemon.json.bak-forensic.Scoring and comparability caveats
extract-elfwas retrieved and re-judged separately from the normal path.Network access is enabledasymmetry: this appeared in the codex arm only, so the arms did not see identical environment statements./opt/maka-agentmount leak: maka's own tree was visible inside the task container. Narrowing it is tracked separately.Security
The provider API key is visible in plaintext in the VM's process list, passed via
docker compose exec -e. No key reached git, logs, or artifacts, but the process-list exposure is real and should be closed before any further run on a shared host.Recurring operational failure worth naming
Four separate times, a cleanup or probe command matched its own command line —
pkill -f maka-harbor-command-scopesandpkill -f run-harness-abboth match the shell running them. Once it killed the remote shell mid-teardown; once it reported a phantom leftover of exactly 1. Anypkill -f//procsweep written against this harness needs the bracket trick or an explicit self-exclusion.Still open
cc-recover-v2is running on93470383with the watchdog in observe-only mode. If its log stays empty and no claude-code cell ends without a reward, fix(headless): bound teardown and mark the output replay #2151 is confirmed in production; if not, there is a hang path we have not found./procsweeps plus PID files. That requires a Harbor/Pier API change.Astro-Han commentedon Aug 5, 2026
Four-arm Terminal-Bench 2.1 run complete — results, and what the harness had to fix to produce them
The four-arm comparison (Maka, Codex, Claude Code, Reasonix) around one DeepSeek V4 Flash model finished over all 89 tasks. Full report and per-task CSV in #2208; the harness changes it required are in #2177, now merged.
Results. Codex 70/89 (78.65%), Maka 65/89 (73.03%), Claude Code 57/89 (64.04%), Reasonix 55/89 (61.80%). The Codex–Maka gap is not significant — 19 discordant pairs split 12/7, exact McNemar p = 0.359. Only Codex over Claude Code (p = 0.0072) and Codex over Reasonix (p = 0.0026) clear 0.05, and both survive Bonferroni. Maka has the lowest cost per pass ($0.03215).
Two independent full runs agree, including on the non-significance. The earlier three-arm run over the same 89 tasks scored Codex 67, Maka 60, Claude Code 57, with Codex–Maka at p = 0.210. Codex is the only arm configured identically in both, and it is the informative one: 19% of its tasks changed outcome between the runs while its net score moved by 3. About a fifth of this suite is decided by run-to-run variation rather than by any property of the harness under test — that is the noise floor a five-task claim has to clear, and it does not.
Where the spread actually comes from. On the 60 tasks where no arm exhausted its budget, the top three invert and land within 3.33 points: Claude Code 90.00%, Codex 88.33%, Maka 86.67%. Claude Code has the best conditional pass rate and the worst headline score because it ran out of budget on 26.97% of the suite — more than twice any other arm. The headline tracks whether an arm finishes, not whether it can solve the task.
Topology decision. All four arms now run inside the task container. Maka previously ran host-side and bridged tool calls in, which made it the one arm compared under a different execution shape. Moving it exposed four defects, all fixed in #2177 — the two worth recording:
[DONE]. Providers put the usage frame at the end of a stream but not after it, so it had usually already arrived. One arm was credited 1,088 output tokens against a true 27,633 — not a visible gap, a wrong number that read as fully metered. 40% of that arm's requests take this path.apt-gettook 16 s or 300 s from the same host depending on which upstream path it landed on, against a 900 s budget. Redirecting to a regional mirror took one Maka cell from a 900 s timeout to a 111 s pass on an identical tool-call count.Contamination. One task is compromised: on
db-wal-recovery, Maka deleted the encrypted WAL itself (opening the DB with thesqlite3CLI triggered a WAL cleanup), then recovered by downloading the fixture from the publicNousResearch/terminal-bench-2dataset after matchingmain.dbhashes. Codex passed the same task and attempted the same class of lookup againstharbor-framework/terminal-bench-2, which 404s. Both pass, so it is a concordant pair: excluding it moves the scores to 64 and 69 and changes neither the gap nor any p-value. Claude Code is clean; the Reasonix hit is a false positive.Detection here only catches explicit benchmark-repo references. All four arms reach the network (Codex most, 430 requests across 52 domains). A stricter pass — comparing fetched response bodies against final artifacts on passing cells — has not been run.
Caveats. Only the Codex arm is configured identically across the two runs; the four-arm configuration itself has one run. The 900 s budget is a material parameter that binds hardest on Claude Code. Reasonix contributes three unscored cells, so its comparisons rest on 86 shared tasks.
refactor(headless): move the benchmark identity out of the build a gr…
refactor(headless): move the benchmark identity out of the build a gr…
refactor(headless): move the benchmark identity out of the build a gr…
refactor(headless): move the benchmark identity out of the build a gr…
refactor(headless): move the benchmark identity out of the build a gr…
fix(headless): separate an arm that exports nothing from an export th…
feat(headless): scan a finished run's trajectories for retrieval
feat(headless): scan a finished run's trajectories for retrieval (#2319)
Astro-Han commentedon Aug 8, 2026
Closing as completed. The synchronized four-arm Terminal-Bench 2.1 run finished, #2177 landed the required harness work, and #2208 records the report, artifacts, and results. Further ablations or reruns should be tracked as their own experiments with frozen variables.