[Agentic Phase 5] Evaluation framework with pydantic-eval

[bashandbone]

Phase 5: Evaluation Framework with pydantic-eval

Parent Epic: #123
Depends On: All prior phases (#124, #125, #126, #127)
Target: v0.3
Risk Level: Low-Medium

Implement comprehensive evaluation framework using pydantic-eval to measure agent performance, pipeline quality, and enable continuous improvement.

Goals

• Agent performance evaluation
• Pipeline quality metrics
• Search result quality assessment
• Continuous improvement infrastructure
• A/B testing capabilities

Background

pydantic-eval provides:

• Standardized evaluation metrics
• Test case management
• Performance benchmarking
• Comparison frameworks
• Result analysis tools

Implementation Checklist

Evaluation Framework Setup

• Add pydantic-eval dependency
• Create evaluation infrastructure
• Define evaluation datasets
  • Synthetic queries with known answers
  • Real-world query samples
  • Edge case scenarios
• Implement evaluation harness

Metrics Definition

• Relevance metrics
  • Precision@k
  • Recall@k
  • MRR (Mean Reciprocal Rank)
  • NDCG (Normalized Discounted Cumulative Gain)
• Agent quality metrics
  • Reasoning correctness
  • Strategy appropriateness
  • Explanation quality
• Pipeline metrics
  • End-to-end latency
  • Cost per query
  • Success rate
  • Failure mode analysis
• User satisfaction metrics
  • Usefulness ratings
  • Response completeness
  • Clarity of explanations

Evaluation Pipelines

• Automated evaluation runs
  • Nightly evaluation jobs
  • Pre-release validation
  • Regression detection
• Manual evaluation workflows
  • Human review interface
  • Annotation tools
  • Feedback collection
• Continuous evaluation
  • Production query sampling
  • Real-time quality monitoring
  • Alert on degradation

Comparison & Analysis

• Baseline comparisons
  • Simple search vs agent-enhanced
  • Different strategies
  • Model comparisons
• A/B testing framework
  • Traffic splitting
  • Statistical significance
  • Winner selection
• Regression analysis
  • Version-to-version comparison
  • Feature impact assessment
  • Performance trends

Result Tracking & Reporting

• Evaluation database
  • Store all evaluation runs
  • Query/result pairs
  • Metrics over time
• Dashboards
  • Quality trends
  • Performance metrics
  • Cost tracking
• Reporting tools
  • Automated reports
  • Alerting on regressions
  • Improvement recommendations

Testing

• Unit tests for evaluation components
• Validation of metrics
• Test data quality checks
• Evaluation pipeline tests

Configuration

• Evaluation schedules
• Metric thresholds
• Alert configuration
• Dataset management
• Sampling strategies

Success Criteria

• Evaluation framework running regularly
• Metrics provide actionable insights
• Regressions detected automatically
• A/B tests guide decisions
• Documentation complete
• Team trained on evaluation tools

Example Evaluation Scenarios

1. Agent Impact Assessment

Hypothesis: Agents improve search relevance
Test: Compare simple search vs agent-enhanced
Metrics: Precision@5, MRR, user satisfaction
Result: Quantified improvement or not

2. Strategy Optimization

Hypothesis: Smart strategy selection reduces latency
Test: Fixed strategy vs adaptive strategy
Metrics: Latency distribution, quality metrics
Result: Identify optimal routing rules

3. Model Comparison

Hypothesis: GPT-4 agents outperform GPT-3.5
Test: Same pipelines, different models
Metrics: Quality, cost, latency
Result: ROI analysis for model selection

4. Data Provider Value

Hypothesis: External context improves answers
Test: With vs without data providers
Metrics: Completeness, accuracy
Result: Determine which providers to use

Integration Points

• Telemetry (Add monitoring: track current file, provider health checks, latency tracking #112): Evaluation feeds into monitoring
• Pipeline Orchestration ([Agentic Phase 4] Pipeline orchestration with pydantic-graph #127): Evaluate different pipeline strategies
• Context Agents ([Agentic Phase 3] Internal context agent for orchestrated search/response #126): Measure reasoning quality

Reference

• pydantic-eval: https://github.com/pydantic/pydantic-eval (when available)
• Related: All agentic phases benefit from evaluation
• Integration: Add monitoring: track current file, provider health checks, latency tracking #112 (monitoring), [Agentic Phase 4] Pipeline orchestration with pydantic-graph #127 (pipeline optimization)

[github-actions]

Claude finished @bashandbone's task —— View job

---

Issue Analysis Complete

I've reviewed this Phase 5 evaluation framework proposal and the broader agentic enhancement initiative. Here's my comprehensive analysis:

---

Summary

This issue proposes implementing a comprehensive evaluation framework using pydantic-evals to measure agent performance, pipeline quality, and enable continuous improvement for CodeWeaver's agentic capabilities. This is Phase 5 of the larger agentic enhancement epic (#123).

Status: Well-specified and ready for implementation, but blocked by dependencies (Phases 1-4 not yet complete).

---

Key Findings

1. pydantic-evals is Available and Well-Documented

✅ Confirmed: pydantic-evals exists as part of the pydantic-ai ecosystem (introduced in v0.0.47, currently in beta)

• Package: Available via pip install pydantic-evals
• Documentation: Comprehensive documentation available at /home/runner/work/codeweaver/codeweaver/data/context/apis/complete_docs/pydantic-evals.md
• Core Features: Dataset, Case, Evaluator, LLMJudge, evaluation reports, and integration with Logfire
• Status: Beta (API subject to change, but actively maintained)

2. Strong Alignment with Project Constitution

This proposal demonstrates excellent constitutional compliance:

• ✅ Principle III (Evidence-Based Development): Evaluation framework enables verifiable evidence for all technical decisions
• ✅ Principle II (Proven Patterns): Leverages established pydantic ecosystem patterns
• ✅ Principle I (AI-First Context): Agents improve through measured performance
• ✅ Principle IV (Testing Philosophy): Focus on effectiveness over coverage metrics

3. Dependencies & Current State

Blocked By (all currently OPEN):

• [Agentic Phase 1] Integrate agents into find_code pipeline #124: [Agentic Phase 1] Integrate agents into find_code pipeline
• [Agentic Phase 2] Integrate data providers into pipeline #125: [Agentic Phase 2] Integrate data providers into pipeline
• [Agentic Phase 3] Internal context agent for orchestrated search/response #126: [Agentic Phase 3] Internal context agent for orchestrated search/response
• [Agentic Phase 4] Pipeline orchestration with pydantic-graph #127: [Agentic Phase 4] Pipeline orchestration with pydantic-graph

Current Project Status:

• Early development phase - core MCP server, CLI, and provider system partially implemented
• Strong testing infrastructure already in place (79 test files, 881 test occurrences)
• Existing pytest markers for categorization (unit, integration, e2e, benchmark, etc.)
• No pydantic-evals dependency in pyproject.toml yet

4. Implementation Difficulty Assessment

Complexity: Medium

Rationale:

• ✅ Clear API with good documentation
• ✅ Existing test infrastructure to build on
• ✅ Well-defined metrics in the issue
• ⚠️ Requires all prior phases (1-4) to be complete for meaningful evaluation
• ⚠️ Beta status means potential API changes
• ⚠️ Requires thoughtful dataset creation (synthetic + real-world scenarios)

Estimated Effort: 2-3 weeks of focused development after dependencies complete

---

Related Issues & Context

Parent Epic: #123 - [EPIC] Agentic enhancements: intelligent search, orchestration, and evaluation

Dependencies: Must complete #124, #125, #126, #127 first

Complementary Work:

• Add monitoring: track current file, provider health checks, latency tracking #112: Monitoring/telemetry (evaluation feeds into observability)
• [EPIC] Replace registries with dependency injection system #116: DI System (enables agent injection for evaluation)
• Implement provider system for reranker models (mirror embedding system) #109: Reranker System (quality improvements work together)

Existing Documentation:

• /data/context/apis/complete_docs/pydantic-evals.md - Full pydantic-evals docs (744 lines)
• /data/context/apis/pydantic-graph-and-evals.md - Integration research
• /data/context/apis/pydantic-ai.md - Agent framework context

---

Recommendations

1. Label Suggestions

Appropriate labels for this issue (many already exist in the repo):

• ✅ evaluation (exists)
• ✅ pydantic-eval (exists)
• ✅ quality (exists)
• ✅ testing (exists)
• ✅ agents (exists)
• ✅ pipeline (exists)
• ✅ enhancement (standard)
• 🆕 Suggested new: blocked (already exists!) - should be applied since this depends on [Agentic Phase 1] Integrate agents into find_code pipeline #124-127

2. Implementation Approach

Phased Implementation (after dependencies complete):

1. Foundation (Week 1)

   • Add pydantic-evals dependency to pyproject.toml
   • Create evaluation infrastructure in src/codeweaver/evaluation/
   • Define core evaluation datasets (synthetic queries)

2. Metrics & Evaluators (Week 1-2)

   • Implement relevance metrics (Precision@k, Recall@k, MRR, NDCG)
   • Create custom evaluators for agent quality
   • Build pipeline performance metrics

3. Automation & Reporting (Week 2-3)

   • Set up automated evaluation runs
   • Create dashboards for quality trends
   • Implement regression detection

4. Continuous Improvement (Week 3+)

   • A/B testing framework
   • Production query sampling
   • Real-time quality monitoring

3. Risk Mitigation

Beta Status Risk:

• Pin pydantic-evals version in pyproject.toml
• Monitor pydantic-ai releases for breaking changes
• Contribute feedback to pydantic team

Dependency Risk:

• Apply blocked label until [Agentic Phase 1] Integrate agents into find_code pipeline #124-127 complete
• Consider starting dataset creation in parallel (doesn't require agent infrastructure)

4. API Surface Impact

Low Impact - This is primarily internal tooling:

• No changes to user-facing MCP API
• No changes to Human API (CLI/config)
• May expose evaluation results via monitoring dashboards (future)

---

Potential Fixes/Approach

Dataset Creation Strategy

Start building evaluation datasets now (before dependencies complete):

# Example: Create initial dataset for code search evaluation
from pydantic_evals import Case, Dataset

# Synthetic test cases for authentication queries
auth_cases = [
    Case(
        name="auth_middleware_query",
        inputs="find authentication middleware patterns",
        expected_output=["src/auth/middleware.py", "src/middleware/auth.py"],
        metadata={"category": "authentication", "difficulty": "medium"}
    ),
    # ... more cases
]

# Dataset can be created before agent infrastructure exists
code_search_dataset = Dataset(cases=auth_cases)

Integration with Existing Testing

Leverage current pytest infrastructure:

# tests/evaluation/test_search_quality.py
import pytest
from pydantic_evals import Dataset

@pytest.mark.evaluation
@pytest.mark.integration
async def test_search_relevance_metrics():
    """Evaluate search quality using pydantic-evals."""
    # Use existing test fixtures + pydantic-evals evaluators
    ...

---

Conclusion

This is a well-thought-out proposal that strongly aligns with CodeWeaver's constitutional principles. The issue is thorough, the dependency chain is clear, and the implementation path is viable.

Status: ✅ Ready for implementation after #124-127 complete

Recommended Action:

1. Apply blocked label
2. Begin dataset creation in parallel (low-risk preparatory work)
3. Monitor pydantic-evals evolution (currently beta)
4. Implement once Phase 1-4 dependencies are complete

---

Labels to Apply

Suggested labels (most already exist):

• blocked - Depends on [Agentic Phase 1] Integrate agents into find_code pipeline #124-127
• enhancement - New feature
• evaluation - Evaluation framework
• pydantic-eval - Uses pydantic-evals
• quality - Quality improvement
• testing - Testing infrastructure
• agents - Agent evaluation
• pipeline - Pipeline metrics

---