A visual workbench for A/B testing AI skills. Upload two Agent Skill packages or single-file skills, run them through a batch of test prompts, and let an AI judge score the results.
DEMO VIDEO: SkillEval v1.1 product video
I built this to improve Anthropic's frontend design skill and prove the improvements with data. That project was hardcoded to frontend evaluation, so I extracted the core evaluation engine and made it extensible—now you can 1v1 any two skills, in any domain, with any model.
For the full story on how this started and why data-driven skill development matters, see Teaching Claude to Design Better.
Skill Evaluator v1.2.1 refreshes the supported frontier model list with Claude Opus 4.8, newer Gemini options, and Grok 4.3 through xAI. It also removes disabled Anthropic Fable access, migrates older saved Opus defaults to Opus 4.8, and validates saved defaults so unsupported model IDs fall back safely.
Skill Evaluator v1.2 adds support for zipped Agent Skill packages. You can now upload a standard .zip skill package with a root SKILL.md, and Skill Evaluator will preserve its separate references, scripts, assets, and supporting files when generating configs and running evaluations.
Single-file .md and .txt skills still work as before, but internally they are treated as one-file Agent Skill packages so both upload paths use the same evaluation model.
Skill Evaluator v1.1 adds support for current Anthropic, OpenAI, Google, and xAI models, expanding the app beyond Anthropic-only workflows. Model selection now supports provider-specific API keys, cross-provider choices, and separate defaults for skill output generation and judge scoring.
This release also introduces a refreshed design language, with cleaner surfaces, tighter controls, restrained depth, and fully considered light and dark modes.
Configure — Upload two single-file skills or zipped Agent Skill packages (A and B) and set up your evaluation. Click the ✨ Generate button to have AI analyze your skills and create appropriate evaluation criteria and test prompts automatically. Choose your output type (text, visual, or both) based on what your skills produce.
Light:
Dark:
Light:
Dark:
Evaluate — Select your model and run all prompts through both skills simultaneously. Each prompt generates two outputs (one from Skill A, one from Skill B) which you can compare side-by-side. The status badges show progress in real-time.
Light:
Dark:
Results — After generation, run the judge to score both outputs against your criteria. The summary view shows overall wins/losses, while the detailed breakdown reveals per-criterion scores and patterns.
Light:
Dark:
Light:
Dark:
Settings — Manage provider keys for Anthropic, OpenAI, Gemini, and xAI, then set default models for configuration, generation, and judging.
Light:
Dark:
- Node.js 18+
- Provider key for at least one supported provider:
- Anthropic (console.anthropic.com)
- OpenAI (platform.openai.com)
- Gemini (aistudio.google.com)
- xAI (console.x.ai)
git clone https://github.com/justinwetch/SkillEval.git
cd SkillEval
npm install
npm --prefix app installnpm run dev:appOpen http://localhost:5173 in your browser.
- Go to Configure and choose the Output model and Judge model
- If a selected model needs provider access, click its Add Key action to open the right Settings panel
- Upload two single-file skills or zipped Agent Skill packages
- Click Generate All from Skills to auto-generate criteria and prompts
- Go to Evaluate and click Run All Evals
- Once generation completes, click Judge All to score the outputs
- Review the results in Summary and Detailed Breakdown tabs
Choose models deliberately for the two major phases. The Output model runs both skills, while the Judge model generates criteria/prompts and scores the results.
| Model | Provider | Characteristics |
|---|---|---|
| Claude Opus 4.8 | Anthropic | Strong model for criteria, prompts, and judging (default judge) |
| Claude Sonnet 4.6 | Anthropic | Balanced capability and speed (default output generation) |
| Claude Haiku 4.5 | Anthropic | Fast, cost-effective, good for iteration |
| GPT-5.5 | OpenAI | Frontier model for complex reasoning and coding |
| GPT-5.4 Mini / Nano | OpenAI | Lower-latency, lower-cost OpenAI options |
| Gemini 3.5 Flash | Gemini | Stable Gemini model for sustained coding and agentic tasks |
| Gemini 3.1 Pro Preview | Gemini | Advanced Gemini Pro option |
| Gemini 3 Flash Preview / 3.1 Flash-Lite | Gemini | Newer Gemini speed/capability options for agentic and lightweight workloads |
| Grok 4.3 | xAI | xAI flagship text and image-input model via the Responses API |
For judging, Claude Opus 4.8 is the default because strong reasoning is useful for criteria, prompts, and nuanced scoring. Claude Sonnet 4.6 is the default for skill output generation. Anthropic, OpenAI, Gemini, and xAI models are available in the same selectors when their provider keys are configured.
For documentation on how to write effective skill files, see Claude Code Skills.
Skill Evaluator accepts:
.zipAgent Skill packages with a rootSKILL.md.mdsingle-file skills.txtsingle-file skills
The test-skills/ folder contains example skill files you can use as reference:
sql-skill-a.md— Basic SQL query generation skillsql-skill-b.md— Advanced SQL skill with optimization focuszip-fixtures/— Example zipped Agent Skill packages
For visual evaluations (HTML/CSS skills that produce rendered output), you'll need the local runner:
# From the project root
npm run serverThe server runs on port 3001 and uses local Chrome through Puppeteer to capture screenshots of rendered HTML. If you're evaluating text-only skills (code, SQL, writing, etc.), you don't need this.
SkillEval/
├── app/ # React application
│ ├── src/
│ │ ├── components/ # UI components
│ │ ├── contexts/ # React context (config, run state, settings)
│ │ ├── utils/ # Core logic (API, eval, judging)
│ │ └── views/ # Page components
│ └── package.json
├── test-skills/ # Example skill files
├── screenshots/ # GUI screenshots
└── screenshot-server.js # Optional local runner for visual evaluation
Contributions welcome! Open an issue for bugs, feature requests, or to share interesting skill files you've tested.
MIT
Built by Justin Wetch