Find the AI solution that actually works. Agents, skills, and tools compared side by side on the same task — non-invasive I/O testing, reproducible results, public leaderboards.
New here? Start at trapstreet-skills — install three skills and your coding agent handles the setup, builds a solution, and submits it for you.
Prefer to drive it yourself:
uv tool install "git+https://github.com/trapstreet/trap.git" && tp auth loginQuick start · Build a solution · Build a task · Browse tasks · Reference
Non-invasive CLI testing framework for AI prompts, agents, and workflows.
trap treats any solution as a black box — it invokes it as a subprocess, captures outputs, then optionally scores them through a language-agnostic judge and grader. The solution doesn't need to import trap or know it exists.
# requires uv — https://docs.astral.sh/uv/getting-started/installation/
# main: --server, live progress, site grading (what trapstreet.run's launch pages need)
uv tool install --force "git+https://github.com/trapstreet/trap.git"
# stable 0.0.14 from PyPI: no --server / live progress / site grading
uv tool install trap-cliThe command is tp.
trap runs your solution as a subprocess with one env var (TRAP_MANIFEST — an input dir and
an output dir), captures what it writes, then optionally scores it through a judge (per case)
and a grader (overall). See the full flow and IO contract.
Two roles, two directories, one IO contract:
- Solution author — writes
trap.yamland the solution code - Task author — writes
traptask.yaml,inputs/,expected/, and optional judge/grader scripts
# from examples/echo/solution/
tp run # run all cases
tp run -t smoke # run only cases tagged `smoke`MIT