The console script is task. In a checkout, invoke it as uv run task <command>.
Behavior shared by every subcommand:
--jsonswitches the output to a single machine-readable JSON payload with no table decoration. An invocation is either fully human-readable or fully JSON, never mixed.- Exit codes carry meaning:
0for success or a resolved run,1for a contract not met (an unresolved run, an unknown run id),2for an error (a broken bundle, a container failure). Expected failures print a one-line cause, not a traceback. - Every invocation, including failed ones, writes an audit row to the local SQLite store.
Materialize a bundle's image and run a build-verification smoke check.
| Option | Meaning |
|---|---|
--from-dataset |
Ingest a SWE-bench Pro dataset row instead of reading a local bundle. |
--offset <n> |
Which dataset row to fetch. |
--dest <path> |
Where to write the materialized bundle. |
--no-smoke |
Skip build verification. |
Two modes. Given a local BUNDLE_PATH, it builds the bundle's Dockerfile or pulls its prebuilt image reference, pins the digest, and smoke-checks that the selected test files are present at the base commit. Given --from-dataset, it fetches one row, converts it into a bundle at --dest, and resolves the row's prebuilt image.
Execute the baseline before any solver touches the code, and assert the full contract explicitly: every pass-to-pass test passes and every fail-to-pass test fails. A baseline that does not hold is reported, not silently accepted.
| Option | Meaning |
|---|---|
--with-gold |
Also apply the golden patch in a fresh container and prove it resolves the task. |
Solve and evaluate the bundle with the selected solver, then grade and persist the run.
| Option | Meaning |
|---|---|
--solver <null|golden|sabotage|agent> |
Which solver attempts the fix. Default null. |
--sabotage-target <path> |
For the sabotage solver: which file to break. |
--network <policy> |
Override the eval phase's network policy. |
The run is a two-container flow. The solve container gets a sanitized repository (git history cut at the base commit, graded tests absent) and produces a patch. A fresh eval container applies that patch, injects the hidden tests, runs the test command, and grades the transitions into four buckets: fail_to_pass, pass_to_pass, fail_to_fail, pass_to_fail. The run is resolved only if every declared test lands in the first two buckets. Artifacts land under runs/<run_id>/: solver.diff, eval.log, report.json, withheld_paths.txt, manifest.json.
The three stub solvers are deterministic oracles that need no credentials. The agent solver requires the llm extra and model credentials in the environment.
List past runs, newest first: run id, bundle, resolved verdict, timestamp.
Show one run in full: the run row (pinned image digest, resolved config), the per-test results with their buckets, the phase timeline, and the artifact list.