Support CBORG in judges.py

[valerie-autumn-skye]

The current version connects to the Anthropic API. This class could be made more generic by adding the API URL and model as a configurable parameter.

Originally posted by @ct-parker in #16 (comment)

[github-actions]

Issue Analysis: Support CBORG in judges.py

Context and Background

This issue stems from PR #16 discussion about making the evaluation system more generic. Currently, the evaluation framework in metacoder uses DeepEval metrics that are hardcoded to connect to the Anthropic API for LLM-based evaluation (like the GEval metric in runner.py:65-82).

What is CBORG?

CBORG is Lawrence Berkeley National Laboratory's multi-model AI portal that provides:

• Unified access to multiple AI models (ChatGPT 4, Claude, Gemini, Grok, Llama, Qwen)
• API access for developers with personal API keys
• Local and cloud model infrastructure
• Scientific workflow integration

Current Issue

The evaluation system in src/metacoder/evals/runner.py currently uses DeepEval's GEval metric (lines 65-82) which:

1. Hardcodes Anthropic API: The GEval metric internally uses a specific LLM provider (typically Anthropic Claude)
2. Limited configurability: Cannot easily switch to CBORG's unified API endpoint
3. Lacks provider flexibility: No mechanism to specify alternative API URLs or model configurations for judges

Proposed Solution

Make the evaluation system provider-agnostic by:

1. Add API configuration parameters to EvalDataset model in eval_model.py:

   class EvalDataset(BaseModel):
       # ... existing fields
       judge_api_url: Optional[str] = Field(
           default=None, 
           description="Custom API URL for LLM judges (e.g., CBORG endpoint)"
       )
       judge_model: Optional[str] = Field(
           default=None,
           description="Model name for judge evaluation"
       )
       judge_api_key: Optional[str] = Field(
           default=None,
           description="API key for judge evaluation service"
       )

2. Enhance metric instantiation in runner.py to pass custom configurations:

   def get_default_metrics(api_url=None, model=None, api_key=None) -> Dict[str, BaseMetric]:
       return {
           "CorrectnessMetric": GEval(
               name="Correctness",
               criteria="...",
               evaluation_steps=[...],
               threshold=0.8,
               # Add custom API configuration
               model=model or "claude-3-sonnet-20240229",
               api_base=api_url,  # Custom endpoint like CBORG
               api_key=api_key
           )
       }

3. Update runner logic to propagate configuration to metrics

Benefits

• Multi-provider support: Can use CBORG, Anthropic, OpenAI, or other API endpoints
• Cost optimization: Leverage CBORG's potentially more cost-effective access
• Berkeley Lab integration: Native integration with lab infrastructure
• Research reproducibility: Consistent model access across lab projects

Technical Considerations

• Ensure DeepEval supports custom API endpoints for GEval metrics
• Handle authentication differences between providers
• Maintain backward compatibility with existing evaluation configs
• Consider rate limiting and error handling for different providers

This enhancement would make metacoder's evaluation framework truly provider-agnostic and better suited for research environments like Berkeley Lab that have access to multiple AI service providers.

Related Context

From the search results, I can see CBORG is actively used across Berkeley Lab projects for:

• OntoGPT extractions of IJSEM abstracts (issue #174 in berkeleybop/metpo)
• Integration with various bioinformatics workflows
• Cost-effective access to multiple LLM providers (~$0.10 per abstract processing)

This feature would align with the broader Berkeley Lab ecosystem's use of CBORG for scientific AI applications.