Repository navigation
bench: same-round full-suite prompt ablation (89 tasks × 2 arms) to quantify the prompt package's share of the TB2.1 gap #1864
Description
Background
A prompt-swap ablation on Terminal-Bench 2.1 (deepseek-v4-flash-max) produced positive but borderline evidence that Maka's 4-line system prompt package helps OpenCode: on 4 preselected natural-failure tasks, the package went 10/16 vs 6/16 pass (6 treatment-only vs 2 original-only passes, net +4 — exactly at the pre-registered screen threshold, ≈1.41σ, exact sign test p=0.14). The swap was canary-verified single-variable on the wire (only agent.build.prompt differs; #1855). The screen's go-gate says: proceed to confirmation.
Proposed run
89 TB2.1 tasks × 2 arms (opencode-original + opencode-makaprompt) × 1 round = 178 cells, same-round paired (~$14, 4–5h at 4-way Docker parallelism; driver/configs exist from the v1 ablation). Same-round pairing is required: a 15-cell drift control has SE ≈ 13pp — the same magnitude as the effect — so cross-round pairing against v7 cannot support a headline number. v7 comparison is retained as a post-hoc drift observation only.
Pre-registered reads
- Confirmation: net discordance across the 89 pairs, including a free 5th rep on the 4 v1 V-stratum tasks.
- Headline: the package's pp contribution vs the same-round original arm, and its share of the gap (denominator: same-round Maka vs original if a Maka arm is added later; otherwise stated relative to the historical v7 +13.48pp with the caveat).
- T/L strata (first test): does the shorter prompt marginally rescue wall-clock deaths? v1 showed the prompt changes work volume (steps +27%), so the direction is genuinely unknown.
- Harm check: net on tasks outside the v7 MO set should be ≈0.
Sentence-level decomposition is explicitly deferred: splitting a ~1.4σ package effect into thirds would need ~9× the reps for equal power.
Depends on
- feat(headless): allow OPENCODE_CONFIG override for prompt ablation arms #1855 (
MAKA_OPENCODE_CONFIG_PATH+opencode-benchmark-makaprompt.json)
Activity
Astro-Han commentedon Aug 5, 2026
Evidence from the Terminal-Bench 2.1 four-arm run (89 tasks, deepseek-v4-flash, effort max) that bears directly on what this ablation should measure.
The size of the gap. Maka's headless system prompt is 253 characters (
packages/headless/src/system-prompts.ts:12-17). Codex'sbase_instructionsis 17,730 characters, identical across all 89 rollouts in that run.What is not the explanation, so the ablation does not chase it. Two independent verification passes over the Codex source and the archived traces established that Codex's higher-level instruction machinery was entirely inactive in this benchmark:
prompts/templates/goals/continuation.mdis injected only when an active goal exists;codex execnever creates one.codex reviewor the review sub-agent, neither of which the benchmark path uses.update_planis registered, but the function output returned to the model is the fixed string"Plan updated"— the plan itself is emitted as a TUI event and never re-enters model context. It was called 26 times across 89 tasks.get_context_remainingandnew_context_windoware gated onFeature::TokenBudget,default_enabled: false; trace confirms zero calls.Neither harness has any enforced final verification step. The difference that remains live is the plain instruction text itself — which is what this issue proposes to measure.
A behavioral signal worth an outcome column. Against the both-pass baseline (56 tasks, tool calls 26 vs 28), on the 12 tasks Codex passed and Maka did not, Codex's tool calls rose to 60 (×2.14) with reasoning per call roughly flat (×1.04), while Maka's calls stayed at 26 (×1.00) with reasoning per call rising to 655 (×1.93). If the instruction layer is doing work here, an ablation arm should move tool-call count on hard tasks, not just pass rate.
Two confounders to control for, since both were measured in the same run and would otherwise be attributed to the prompt: #2241 (reasoning not round-tripped on the
openai-chatpath, reasoning per request climbing 53 → 428 across step buckets) and the pre-#2028ArchiveReaddefect, which was live during this run and left 38/89 tasks re-reading archived tool output. Run the ablation on post-#2028mainwith #2241 resolved, or the prompt arm absorbs both effects.Astro-Han commentedon Aug 23, 2026
Closing as superseded. The Headless prompt package,
MAKA_OPENCODE_CONFIG_PATH, and the prompt-swap configuration this run depends on were removed by the Eval architecture cutover. The historical 253-character Headless prompt is no longer the prompt used by current Maka execution.A prompt ablation may still be useful, but it needs a newly frozen current Runtime Host request boundary and current Eval cell design rather than reusing these historical arms and baselines.
Triage and wording were prepared with OpenAI Codex and reviewed by the maintainer.