Config

1. apps/eval/src/config.json5, this file must be created locally, see Config Parameters.

{
  "models": ["claude-3-5-sonnet-20241022", "openai/gpt-4o"],
  "agentMode": "local"
}

2. apps/eval/.env, this file must be created locally

ANTHROPIC_API_KEY=xxx
OPENROUTER_API_KEY=xxx
OPENAI_API_KEY=xxx
DOUBAO_1_5_ENDPOINT=xxx
DOUBAO_API_KEY=xxx

XXX_API_KEY is used in apps/eval/src/model.json

{
"models": [
  {
    "model": "claude-3-5-sonnet-20241022",
    "provider": "anthropic",
    "apiKey": "{{ANTHROPIC_API_KEY}}",
    "title": "claude-3-5-sonnet-20241022"
  },
  {
    "title": "openai/o1-preview",
    "provider": "openrouter",
    "model": "openai/o1-preview",
    "apiBase": "https://openrouter.ai/api/v1",
    "apiKey": "{{OPENROUTER_API_KEY}}"
  },
  {
    "title": "gpt-4o",
    "provider": "openai",
    "model": "gpt-4o",
    "apiBase": "https://api.openai.com/v1/",
    "apiKey": "{{OPENAI_API_KEY}}"
  },
  {
    "title": "doubao-pro-1.5-32k",
    "provider": "doubao",
    "model": "{{DOUBAO_1_5_ENDPOINT}}",
    "apiKey": "{{DOUBAO_API_KEY}}"
  }
]
}

For Agent Developer

If you want your own agent for evaluation, detail in Custom Agent.