coding-agent-eval-harness

coding-agent-eval-harness runs deterministic task fixtures for coding-agent workflows.

Fixture files can be JSON or simple YAML. Each task runs commands in a temporary work directory, then scores expected exit codes, files, and stdout substrings.

0.1.2 Highlights

• Fixtures can now assert file contents with expect.file_contains.
• A single expected file may require one or many substrings, making generated reports and code edits easier to verify.

JSON Fixture Example

{
  "name": "demo",
  "tasks": [
    {
      "name": "write file",
      "commands": [
        {"args": ["python", "-c", "from pathlib import Path; Path('done.txt').write_text('ok')"]}
      ],
      "expect": {
        "exit_code": 0,
        "files": ["done.txt"],
        "file_contains": {"done.txt": "ok"}
      }
    }
  ]
}

Usage

python -m coding_agent_eval_harness fixture.json
python -m coding_agent_eval_harness fixture.yaml --json
python -m coding_agent_eval_harness fixture.json --source ./starter-repo

Installed script:

coding-agent-eval-harness fixture.json --json

Development

python -m unittest discover -s tests

No network calls are required by the test suite.