Skip to content

joinalahmed/skilleval

BranchesTags

Repository files navigation

SkillEval

License Python Version

100% deterministic evaluation framework for AI agent skills

Open-source, vendor-neutral framework for evaluating AI agent skills with dual-phase scoring, comprehensive security scanning, and transparent grading.

Features

  • ✅ 100% Deterministic - No LLM judges, reproducible results
  • 🛡️ 82+ Security Patterns - OWASP Web/API/LLM/Agentic coverage
  • 📊 Dual-Phase Scoring - Separate packaging quality from runtime effectiveness
  • ⚡ Fast - Phase 1 evaluation in <1 second
  • 🎯 Confidence Weighting - Smart false positive reduction
  • 🐳 Container Isolation - Safe execution with Podman
  • 📈 Baseline Comparison - WITH-SKILL vs WITHOUT-SKILL differential scoring
  • 🎓 A-F Grading - Clear publish decisions with auto-reject

Quick Start

Installation

pip install -r requirements.txt
pip install -e .

Evaluate a Skill

# Fast packaging & security check (<1s)
python3 evaluate_skill.py /path/to/skill --phase1-only

# Full evaluation with runtime testing
python3 evaluate_skill.py /path/to/skill --full

# JSON output for CI/CD
python3 evaluate_skill.py /path/to/skill --phase1-only --format json

Example Output

Total Score: 91.0/100
Grade: A
Publish Decision: APPROVE

PILLAR 1: STATIC TESTS (50 points)
Score: 49.0/50 (Grade A)
  ✅ Frontmatter valid
  ✅ Description quality high
  ⚠️  Only 1 eval case (need 3+)

PILLAR 2: SECURITY (50 points)
Score: 42.0/50 (Grade B)
  ✅ No CRITICAL findings
  ⚠️  3 MEDIUM findings (confidence-weighted)

✅ APPROVED - Ready to publish
   Eligible for featured listing

What's Evaluated

Phase 1: Packaging & Security (0-100 points)

Static Tests (50 points)

  • ✅ Frontmatter validity - YAML, name, description, version
  • ✅ Description quality - Length, vocabulary, trigger language
  • ✅ File completeness - SKILL.md, artifacts, structure
  • ✅ Script quality - Python/shell syntax validation
  • ✅ Eval suite - Test case coverage and diversity
  • ✅ Instruction clarity - Code examples, documentation

Security (50 points)

  • 🔒 18 built-in checks (Layer 1) - Zero dependencies
    • Hardcoded secrets (API keys, tokens, passwords)
    • Code injection (SQL, XSS, command injection)
    • Data exfiltration patterns
    • Prompt injection detection
    • OWASP ASI compliance
  • 🔒 64 advanced checks (Layer 2, optional) - SkillSpector integration
    • AST behavioral analysis
    • CVE database lookup
    • YARA malware signatures

Confidence Weighting:

  • ≥ 0.7: Full penalty, CRITICAL = auto-reject
  • 0.5-0.69: Full penalty, normal scoring
  • 0.3-0.49: Advisory only (shown, zero score impact)
  • < 0.3: Hidden

Phase 2: Runtime Effectiveness (0-100 points)

Functional Correctness (50 points)

  • Baseline vs skill comparison (WITH-SKILL - WITHOUT-SKILL)
  • 6 deterministic graders:
    • file_exists - File creation
    • content_match - Regex patterns
    • json_schema - JSON validation
    • command_output - Command results
    • exit_code - Script success
    • line_count - File size verification

LLM Safety (50 points)

  • Unbounded planning detection (>15 turns)
  • Infinite loop detection (high tool/turn ratio)
  • Context rot detection (>200K tokens)
  • Hallucination detection (claims without evidence)
  • Cost tracking (>$0.10 threshold)
  • Efficiency analysis (<20% tool usage)

Grade Scale

Grade Score Decision Meaning
A 90-100 APPROVE (featured eligible) Excellent
B 80-89 APPROVE Good
C 70-79 CONDITIONAL Acceptable with advisory
D 60-69 REQUIRE_ACK Needs improvement
F 0-59 BLOCK Not ready

Auto-Reject Conditions:

  • CRITICAL security finding (confidence ≥ 0.7)
  • Security score < 25/50 (50% floor)

OWASP Coverage

✅ OWASP Top 10 Web (2021) - A01, A03, A06
✅ OWASP API Security (2023) - API1-API10
✅ OWASP LLM Top 10 (2023) - LLM01-LLM09
✅ OWASP Agentic AI (2026) - ASI01-ASI10

Total: 28 unique security patterns

Architecture

Dual-Phase Design

┌─────────────────────────────────────┐
│  Phase 1: Packaging & Security      │
│  ─────────────────────────────      │
│  • Static Tests (50 pts)            │
│  • Security Scan (50 pts)           │
│  • <1 second                        │
│  • No LLM calls                     │
└─────────────────────────────────────┘
              ↓
┌─────────────────────────────────────┐
│  Phase 2: Runtime Effectiveness     │
│  ───────────────────────────────    │
│  • Functional (50 pts)              │
│  • LLM Safety (50 pts)              │
│  • 30-600 seconds                   │
│  • Container isolation              │
└─────────────────────────────────────┘
              ↓
        Dual-Score Report
   Phase 1: 91/100 (A)
   Phase 2: 63/100 (C)
   Overall: 77/100 (B)

Directory Structure

skilleval/
├── src/skilleval/
│   ├── models_phase1.py         # Phase 1 data models
│   ├── models_phase2.py         # Phase 2 data models
│   ├── scorers/
│   │   ├── static_scorer.py     # ST-1 through ST-8
│   │   ├── security_scorer.py   # Layer 1 + Layer 2
│   │   ├── harness_scorer.py    # Functional + Safety
│   │   └── phase1_orchestrator.py
│   ├── pillars/
│   │   ├── static_tests.py
│   │   ├── security.py
│   │   ├── owasp_llm.py
│   │   └── harness.py
│   └── utils/
│       ├── container_executor.py
│       ├── trace_analytics.py
│       └── cve_scanner.py
│
├── tests/                       # Test suite (85% coverage)
├── examples/                    # Example skills
├── docs/                        # Documentation
└── evaluate_skill.py            # Main CLI

CLI Usage

Basic Commands

# Phase 1 only (fast, <1s)
python3 evaluate_skill.py /path/to/skill --phase1-only

# Full evaluation
python3 evaluate_skill.py /path/to/skill --full

# JSON output
python3 evaluate_skill.py /path/to/skill --phase1-only --format json

# Save to file
python3 evaluate_skill.py /path/to/skill --output report.json

Batch Evaluation

for skill in /path/to/skills/*; do
    python3 evaluate_skill.py "$skill" --phase1-only \
        --output "reports/$(basename $skill).json"
done

CI/CD Integration

# Exit code 0 = passed, 1 = failed
python3 evaluate_skill.py /path/to/skill --phase1-only || exit 1

GitHub Actions:

- name: Evaluate Skill
  run: |
    pip install -r requirements.txt
    python3 evaluate_skill.py ./skills/my-skill --phase1-only --format json

Use Cases

Development

  • ✅ Pre-commit quality validation
  • ✅ Security scanning before publication
  • ✅ Interactive feedback during development

CI/CD Pipelines

  • ✅ Automated quality gates
  • ✅ Regression detection
  • ✅ Compliance enforcement

Skill Registries

  • ✅ Publication approval workflow
  • ✅ Featured listing eligibility
  • ✅ Security compliance verification

Security Audits

  • ✅ Vulnerability scanning
  • ✅ OWASP compliance reporting
  • ✅ Secret detection

Example Skill Evaluation

Input: /path/to/jira-comment-poster

Phase 1 Results:

Score: 91/100 (Grade A)
Static: 49/50
Security: 42/50
Duration: 0.01s
Findings: 3 scoreable, 4 advisory
Decision: APPROVED (featured eligible)

Phase 2 Results:

Score: 63/100 (Grade C)
Functional: 0/50 (no graders matched)
Safety: 53/100
Issues: 5 infinite loops, 1.77M tokens
Duration: 679s (11m 20s)
Cost: $0.29

Overall: Grade B (77/100) - APPROVE

Performance

Metric Phase 1 Phase 2 Full
Duration <1s 30-600s 30-600s
Memory <50 MB <512 MB <512 MB
LLM Calls 0 2 per eval case 2 per eval case
Determinism 100% 100% 100%
Throughput 200+ skills/sec 0.1 skills/sec 0.1 skills/sec

Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Development Setup

# Clone repository
git clone https://github.com/skilleval/skilleval.git
cd skilleval

# Install dependencies
pip install -r requirements.txt
pip install -r requirements-dev.txt

# Install in development mode
pip install -e .

# Run tests
pytest tests/

# Run linting
ruff check src/
mypy src/

Adding New Checks

Static Test:

# src/skilleval/scorers/static_scorer.py
def _st9_new_check(self) -> StaticTestResult:
    """ST-9: New static check."""
    # Implementation
    return StaticTestResult(...)

Security Check:

# src/skilleval/scorers/security_scorer.py
def _l1_19_new_check(self, content: str, file: str) -> List[SecurityFinding]:
    """L1-19: New security check."""
    findings = []
    # Pattern matching
    return findings

Documentation

Roadmap

✅ v1.0.0 (Current)

  • Phase 1: Static Tests + Security (production-ready)
  • CLI with JSON/text output
  • Dual-score reporting
  • 82+ security patterns
  • Confidence weighting

🔄 v1.1.0 (In Progress)

  • Phase 2: Runtime Effectiveness integration
  • Harness execution orchestration
  • End-to-end dual-score testing

📋 v2.0.0 (Future)

  • Layer 2 security (SkillSpector)
  • HTML report generation
  • Batch evaluation dashboard
  • MCP server integration

License

MIT License - see LICENSE for details.

Copyright (c) 2026 SkillEval Contributors

Support

References

Status: ✅ Production Ready
Version: 1.0.0
Last Updated: 2026-06-25

About

100% deterministic evaluation framework for AI agent skills with dual-phase scoring, comprehensive security scanning, and transparent grading

github.com/joinalahmed/skilleval

Topics

python testing security ai skills evaluation owasp deterministic agents skill-evaluation llm agent-skills

Resources

Readme

License

MIT license

Contributing

Contributing
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

1 tags

Packages

 
 
 

Contributors

Languages