feat(eval): Chat Eval Pipeline — Swiss Cheese 3-Tier Grader

[mangowhoiscloud]

Summary

• Swiss Cheese 3-Tier Grader 구현 (L1 Code + L2 LLM BARS + L3 Calibration CUSUM)
• Clean Architecture 4-Layer 구조: Domain → Application → Infrastructure (LangGraph)
• 65 files, +6,789 lines, 108 unit tests (100% pass)

Architecture

L1 Code Grader (deterministic, 6 slices)
  ↓ priority_preemptive_reducer
L2 LLM Grader (BARS 5-axis, Self-Consistency, SDK Structured Output)
  ↓ asymmetric weighted sum
L3 Calibration Monitor (Two-sided CUSUM, k=0.5, h=4.0)
  ↓ EvalGrade (A/B/C) + regeneration decision

Commits (7)

#	Commit	Scope
1	feat(eval): Domain layer	EvalGrade, AxisScore, ContinuousScore, CalibrationSample, EvalScoringService
2	feat(eval): Application layer — DTOs, Ports, Exceptions	EvalConfig, EvalResult, 4 Ports, eval exceptions
3	feat(eval): Application Services + Command	CodeGrader, LLMGrader, ScoreAggregator, CalibrationMonitor, EvaluateResponseCommand
4	feat(eval): Infrastructure — BARS evaluator + prompts	OpenAIBARSEvaluator adapter, Pydantic schemas, 6 BARS rubric prompts
5	feat(eval): Infrastructure — LangGraph eval subgraph	eval_node, eval_graph_factory, state/contracts, main graph integration
6	test(eval): Unit tests + pytest markers	108 tests across all layers, eval_unit/eval_regression/eval_capability markers
7	docs(eval): Implementation report	Phase 1+2 report with expert review scores (97.1/100)

Key Design Decisions

• FAIL_OPEN policy: Eval 실패 시 B등급(65.0점), regeneration 미트리거
• Self-Consistency: 경계 점수(2, 4)에서 N회 재평가 후 중앙값 채택
• Bias mitigation: 축 순서 랜덤 셔플(positional bias 완화)
• Cost guardrail: eval_max_cost_usd 초과 시 LLM Grader 스킵
• SDK Structured Output: OpenAI/Gemini 모두 SDK 레벨 constrained decoding 사용

Phase 3 (Next)

• Gateway adapters (PostgreSQL, Redis)
• DI wiring (setup/dependencies.py)
• Integration tests

Test plan

• pytest -m eval_unit — 108 tests, 100% pass
• black --check — clean
• ruff check — clean
• CI pipeline 실행 (PR merge 전)
• Docker 환경에서 전체 테스트 스위트 실행

🤖 Generated with Claude Code